A Q&A with AI regulator Ed Newton-Rex

Dom Aversano

Ed Newton-Rex - photo by Jinnan Wang

In November last year, Ed Newton-Rex, the head of audio at Stability AI, left the company citing a small but significant difference in his philosophy towards training large language models (LLMs). Stability AI was one of several companies that responded to an invitation from the US Copyright Office for comments on generative AI and copyright, submitting an argument that training their models on copyrighted artistic works fell under the definition of fair use: a law which permits the use of copyrighted works for a limited number of purposes, one of which is education. This argument has been pushed by the AI industry more widely, who contest that much like a student who learns to compose music by studying renowned composers, their machine learning algorithms are conducting a similar learning process.

Newton-Rex did not buy the industry’s arguments, and while you can read his full arguments for resigning in his X/Twitter post, central to his argument was the following passage:

(…) since ‘fair use’ wasn’t designed with generative AI in mind — training generative AI models in this way is, to me, wrong. Companies worth billions of dollars are, without permission, training generative AI models on creators’ works, which are then being used to create new content that in many cases can compete with the original works. I don’t see how this can be acceptable in a society that has set up the economics of the creative arts such that creators rely on copyright.

It is important to make clear that Newton-Rex is not a critic of AI; he is an enthusiast who has worked in the machine learning field for more than a decade; his contention is narrowly focused on the ethics surrounding the training of AI models.

Newton-Rex’s response to this was to set up a non-profit called Fairly Trained, which awards certificates to AI companies whose training data they consider ethical.

Their mission statement contains the following passage:

There is a divide emerging between two types of generative AI companies: those who get the consent of training data providers, and those who don’t, claiming they have no legal obligation to do so.

In an attempt to gain a better understanding of Newton-Rex’s thinking on this subject, I conducted a Q&A by email. Perhaps the most revealing admission is that Newton-Rex desires to eliminate his company. What follows is the unedited text. 

Fairly Trained is a non-profit founded by Ed Newton-Rex that award certificates to AI companies who train their models in a manner that is deemed ethical.

Do you think generative artificial intelligence is an accurate description of the technology Fairly Trained certifies?

Yes!

Having worked inside Stability AI and the machine learning community, can you provide a sense of the culture and the degree to which the companies consider artists’ concerns?

I certainly think generative AI companies are aware of and consider artists’ concerns. But I think we need to measure companies by their actions. In my view, if a company trains generative AI models on artists’ work without permission, in order to create a product that can compete with those artists, it doesn’t matter whether or not they’re considering artists’ concerns – through their actions, they’re exploiting artists.

Many LLM companies present a fair use argument that compares machine learning to a student learning. Could you describe why you disagree with this?

I think the fair use argument and the student learning arguments are different.

I don’t think generative AI training falls under the fair use copyright exception because one of the factors that is taken into account when assessing whether a copy is a fair use is the effect of the copy on the potential market for, and value of, the work that is copied. Generative AI involves copying during the training stage, and it’s clear that many generative AI models can and do compete with the work they’re trained on.

I don’t think we should treat machine learning the same as human learning for two reasons. First, AI scales in a way no human can: if you train an AI model on all the production music in the world, that model will be able to replace the demand for pretty much all of that music. No human can do this. Second, humans create within an implicit social contract – they know that people will learn from their work. This is priced in, and has been for hundreds of years. We don’t create work with the understanding that billion-dollar corporations will use it to build products that compete with us. This sits outside of the long-established social contract. 

Do you think that legislators around the world are moving quickly enough to protect the rights of artists?

No. We need legislators to move faster. On current timetables, there is a serious risk that any solutions – such as enforcing existing copyright law, requiring companies to reveal their training data, etc. – will be too late, and these tools will be so widespread that it will be very hard to roll them back.

At Fairly Trained you provide a certification that signifies that a company trains their models on ‘data provided with the consent of its creators’. How do you acquire an accurate and transparent knowledge of the data each company is using?

They share their data with us confidentially.

For Fairly Trained to be successful it must earn people’s trust. What makes your organisation trustworthy?

We are a non-profit, and we have no financial backing from anyone on either side of this debate (or anyone at all, in fact). We have no hidden motives and no vested interests. I hope that makes us trustworthy.

If your ideal legislation existed, would a company like Fairly Trained be necessary? 

No, Fairly Trained would not be necessary. I very much hope to be able to close it down one day!

To learn more about what you have read in this article you can visit the Fairly Trained website or Ed Newton-Rex’s website. 

Dom Aversano is a British-American composer, percussionist, and writer. You can discover more of his work at the Liner Notes.

Back

Getting started with Interactive Machine Learning for openFrameworks – On-demand

Level: Intermediate – C++ required

Using openFrameworks, ofxRapidLib and ofxMaximilian, participants will learn how to integrate machine learning into generative applications. You will learn about the interactive machine learning workflow and how to implement classification, regression and gestural recognition algorithms.

You will  explore a static classification approach that employs the k-Nearest Neighbour (KNN) algorithm to categorise data into discrete classes. This will be followed by an exploration of static regression problems that will use multilayer perceptron neural networks to perform feed-forward, non-linear regression on a continuous data source. You will also explore an approach to temporal classification using dynamic time warping which allows you to analyse and process gestural input

This knowledge will allow you to build your own complex interactive artworks.

By the end of this series the participant will be able to:

Overall:

  • Set up an openFrameworks project for machine learning

  • Describe the interactive machine learning workflow

  • Identify the appropriate contexts in which to implement different algorithms

  • Build interactive applications based on classification, regression and gestural recognition algorithms

Session 1:

  • Set up an openFrameworks project for classification

  • Collect and label data

  • Use the data to control audio output

  • Observe output and evaluate model

Session 2:

  • Set up an openFrameworks project for regression

  • Collect data and train a neural network

  • Use the neural network output to control audio parameters

  • Adjust inputs to refine the output behaviour

Session 3:

  • Set up an openFrameworks project for series classification

  • Design gestures as control data

  • Use classification of gestures to control audio output

  • Refine gestural input to attain desired output

Session 4:

  • Explore methods for increasing complexity

  • Integrate visuals for multimodal output

  • Build mapping layers

  • Use models in parallel and series

Session Study Topics

Session 1:

  • Supervised Static Classification

  • Data Collection and Labelling

  • Classification Implementation

  • Model Evaluation

Session 2:

  • Supervised Static Regression

  • Data Collection and Training

  • Regression Implementation

  • Model Evaluation

Session 3:

  • Supervised Series Classification

  • Gestural Recognition

  • Dynamic Time Warp Implementation

  • Model Evaluation

Session 4:

  • Data Sources

  • Multimodal Integration

  • Mapping Techniques

  • Model Systems

Requirements

  • A computer with internet connection

  • Installed versions of the following software:

    • openFrameworks

    • ofxRapidLib

    • ofxMaxim

  • Preferred IDE (eg. XCode / Visual Studio)

About the workshop leader 

Bryan Dunphy is an audiovisual composer, musician and researcher interested in using machine learning to create audiovisual art. His work explores the interaction of abstract visual shapes, textures and synthesised sounds. He is interested in exploring strategies for creating, mapping and controlling audiovisual material in real time. He is close to completion of his PhD in Arts and Computational Technology at Goldsmiths, University of London.

Back

Max and Machine Learning with RunwayML – On-demand

Level: Intermediate

RunwayML is a platform that offers AI tools to artists without any coding experience. Max/MSP is a visual programming environment used in media art that can be used to control RunwayML in a more efficient way. At the end of the workshop you will be able to train trendy machine learning models and generate videos by walking a latent space through Max and NodeJS.

Session Learning Outcomes

By the end of the course a successful student will be able to:

  • Understand the RunwayML workflow

  • Use Node4Max to control RunwayML and generate a video.

  • Explore ML trendy models

  • Create a Dataset

  • Train a ML model

  • Process videos with the VIZZIE library.

Session 1

– Introduction to the course

– What’s machine learning, deep learning and neural networks?

– What’s RunwayML?

– What’s Max/MSP/Jitter and NodeJS?
– Dataset and models training with RunwayML

Session 2

– What’s a GAN and styleGAN?

– Latent space walk

– Image and video generation with RunwayML, Max and Node4Max (part 1)

Session 3

– Image and video generation with RunwayML, Max and Node4Max (part 2)

Session 4

– processing Images and videos with VIZZIE2 and Jitter.

Session Study Topics

  • Generate images and video through AI

  • Request data to models and save images on your local drive

  • Generate video from images

  • Communication protocols (web sockets and https requests)

  • AI models used in visual art.

  • Video processing

  • Models training

Requirements

  • A computer and internet connection

  • Access to a copy of Max 8 (either trial or licence)

  • A code editor such as Visual Studio Code, Sublime or Atom
  • Attendees need to create a RunwayML account –  https://app.runwayml.com/signup.
    • Upon setting up an account you will receive 10$ credit for free
    • Approx. 50$ credits will be required to complete the course however these do not need to purchased in advance
    • 20% RunwayML discount code will be provided to participant who sign up to the course 

About the workshop leader 

Marco Accardi is a trained musician, multimedia artist, developer and teacher based in Berlin.

He is the co-founder of Anecoica, a collective that organises events combining art, science and new technologies.

Back

Max meetup 20th March – Americas Edition

Date & Time: Saturday 20th March 3pm PST / 6pm EST

Level: Open to all levels

Hosted by Chloe Alexandra & Francisco Botello
 
With presentations by: 
Philip Meyer – Seven Spaces patch
Joaquin Jimenez: Machine Learning on Max with ML.* to create Dub Music
Shomit Barua – Creative Coding: Exercises in Circumnavigation

Overview 

Join the Max meetup to share ideas and learn with other artists, coders and performers. Showcase your patches, pair with others to learn together, get help for a school assignment, or discover new things.  

The meetup runs via Zoom. The main session features short presentations from Max users. Breakout rooms are created on the spot on specific topics, and you can request a new topic at any time. 

In the breakout rooms, you can share your screen to show other participants something you’re working on, ask for help, or help someone else.

Ready to present your work?

Everyone is welcome to propose a presentation. Just fill in this short form and you’ll be put on the agenda on a first come first served basis. 

Presentations should take no more than 5 minutes with 5 minutes Q&A and we’ll have up to 5 presentations at each meetup. 

List of presenters will be announced before each event. 

Requirements 

  • A computer and internet connection

Berlin Code of Conduct

We ask all participants to read and follow the Berlin Code of Conduct and contribute to creating a welcoming environment for everyone.

Back

Immersive AV Composition -On demand / 2 Sessions

Level: Advanced

These workshops will introduce you to the ImmersAV toolkit. The toolkit brings together Csound and OpenGL shaders to provide a native C++ environment where you can create abstract audiovisual art. You will learn how to generate material and map parameters using ImmersAV’s Studio() class. You will also learn how to render your work on a SteamVR compatible headset using OpenVR. Your fully immersive creations will then become interactive using integrated machine learning through the rapidLib library.

Session Learning Outcomes

By the end of this session a successful student will be able to:

  • Setup and use the ImmersAV toolkit

  • Discuss techniques for rendering material on VR headsets

  • Implement the Csound API within a C++ application

  • Create mixed raymarched and raster based graphics

  • Create an interactive visual scene using a single fragment shader

  • Generate the mandelbulb fractal

  • Generate procedural audio using Csound

  • Map controller position and rotation to audiovisual parameters using machine learning

Session Study Topics

  • Native C++ development for VR

  • VR rendering techniques

  • Csound API integration

  • Real-time graphics rendering techniques

  • GLSL shaders

  • 3D fractals

  • Audio synthesis

  • Machine learning

Requirements

  • A computer and internet connection

  • A web cam and mic

  • A Zoom account

  • Cloned copy of the ImmersAV toolkit plus dependencies

  • VR headset capable of connecting to SteamVR

About the workshop leader 

Bryan Dunphy is an audiovisual composer, musician and researcher interested in generative approaches to creating audiovisual art in performance and immersive contexts. His work explores the interaction of abstract visual shapes, textures and synthesised sounds. He is interested in exploring strategies for creating, mapping and controlling audiovisual material in real time. He has recently completed his PhD in Arts and Computational Technology at Goldsmiths, University of London.

Back

Visual Music Performance with Machine Learning – On demand

Level: Intermediate

In this workshop you will use openFrameworks to build a real-time audiovisual instrument. You will generate dynamic abstract visuals within openFrameworks and procedural audio using the ofxMaxim addon. You will then learn how to control the audiovisual material by mapping controller input to audio and visual parameters using the ofxRapid Lib add on.

Session Learning Outcomes

By the end of this session a successful student will be able to:

  • Create generative visual art in openFrameworks

  • Create procedural audio in openFrameworks using ofxMaxim

  • Discuss interactive machine learning techniques

  • Use a neural network to control audiovisual parameters simultaneously in real-time

Session Study Topics

  • 3D primitives and perlin noise

  • FM synthesis

  • Regression analysis using multilayer perceptron neural networks

  • Real-time controller integration

Requirements

  • A computer and internet connection

  • A web cam and mic

  • A Zoom account

  • Installed version of openFrameworks

  • Downloaded addons ofxMaxim, ofxRapidLib

  • Access to MIDI/OSC controller (optional – mouse/trackpad will also suffice)

About the workshop leader 

Bryan Dunphy is an audiovisual composer, musician and researcher interested in generative approaches to creating audiovisual art. His work explores the interaction of abstract visual shapes, textures and synthesised sounds. He is interested in exploring strategies for creating, mapping and controlling audiovisual material in real time. He has recently completed his PhD in Arts and Computational Technology at Goldsmiths, University of London.

Back

An Introduction to Markov Chains: Machine Learning in Max/MSP

Difficulty level: Beginner

Overview

Markov chains are mathematical models that have existed in various forms since the 19th century, which have been used to aid statistical modelling in many real-world contexts, from economics to cruise control in cars. Composers have also found musical uses for Markov Chains, although the implied mathematical knowledge needed to implement them often appears daunting.

In this workshop we will demystify the Markov Chain and make use of the popular ml.star library in Max/MSP to implement Markov Chains for musical composition. This will involve preparing and playing MIDI files into the system (as a form of Machine Learning) and capturing the subsequent output as new MIDI files. By the end of the session you will have the knowledge of how to incorporate Markov Chains into your future compositions at various levels.

Topics

  • Max
  • Markov Chains
  • Machine Learning
  • Algorithmic Composition

Requirements 

  • You should have a basic understanding of the Max workflow and different data types.
  • Knowledge of MIDI format and routing to DAWs (Ableton, Logic etc) would be a plus, although Max instruments will be provided.
  • No prior knowledge of advanced mathematical or machine learning concepts are necessary, the focus will be on musical application.

About the workshop leader

Samuel Pearce-Davies is a composer, performer, music programmer and Max hacker living in Cornwall, UK.

With a classical music background, it was his introduction to Max/MSP during undergraduate studies at Falmouth University that sparked Sam’s passion for music programming and algorithmic composition.

Going on to complete a Research Masters in computer music, Sam is now studying a PhD at Plymouth University in music-focused AI.

Back
About
Privacy