Ask Me Anything: Programmable Lighting
Ask Me Anything: Programmable Lighting
Ask Me Anything on how to design vocalization for creatures with Eduardo Pesole
Ask Me Anything on how to design vocalization for creatures with Eduardo Pesole
A Q&A with AI regulator Ed Newton-Rex
Dom Aversano
In November last year, Ed Newton-Rex, the head of audio at Stability AI, left the company citing a small but significant difference in his philosophy towards training large language models (LLMs). Stability AI was one of several companies that responded to an invitation from the US Copyright Office for comments on generative AI and copyright, submitting an argument that training their models on copyrighted artistic works fell under the definition of fair use: a law which permits the use of copyrighted works for a limited number of purposes, one of which is education. This argument has been pushed by the AI industry more widely, who contest that much like a student who learns to compose music by studying renowned composers, their machine learning algorithms are conducting a similar learning process.
Newton-Rex did not buy the industry’s arguments, and while you can read his full arguments for resigning in his X/Twitter post, central to his argument was the following passage:
(…) since ‘fair use’ wasn’t designed with generative AI in mind — training generative AI models in this way is, to me, wrong. Companies worth billions of dollars are, without permission, training generative AI models on creators’ works, which are then being used to create new content that in many cases can compete with the original works. I don’t see how this can be acceptable in a society that has set up the economics of the creative arts such that creators rely on copyright.
It is important to make clear that Newton-Rex is not a critic of AI; he is an enthusiast who has worked in the machine learning field for more than a decade; his contention is narrowly focused on the ethics surrounding the training of AI models.
Newton-Rex’s response to this was to set up a non-profit called Fairly Trained, which awards certificates to AI companies whose training data they consider ethical.
Their mission statement contains the following passage:
There is a divide emerging between two types of generative AI companies: those who get the consent of training data providers, and those who don’t, claiming they have no legal obligation to do so.
In an attempt to gain a better understanding of Newton-Rex’s thinking on this subject, I conducted a Q&A by email. Perhaps the most revealing admission is that Newton-Rex desires to eliminate his company. What follows is the unedited text.
Do you think generative artificial intelligence is an accurate description of the technology Fairly Trained certifies?
Yes!
Having worked inside Stability AI and the machine learning community, can you provide a sense of the culture and the degree to which the companies consider artists’ concerns?
I certainly think generative AI companies are aware of and consider artists’ concerns. But I think we need to measure companies by their actions. In my view, if a company trains generative AI models on artists’ work without permission, in order to create a product that can compete with those artists, it doesn’t matter whether or not they’re considering artists’ concerns – through their actions, they’re exploiting artists.
Many LLM companies present a fair use argument that compares machine learning to a student learning. Could you describe why you disagree with this?
I think the fair use argument and the student learning arguments are different.
I don’t think generative AI training falls under the fair use copyright exception because one of the factors that is taken into account when assessing whether a copy is a fair use is the effect of the copy on the potential market for, and value of, the work that is copied. Generative AI involves copying during the training stage, and it’s clear that many generative AI models can and do compete with the work they’re trained on.
I don’t think we should treat machine learning the same as human learning for two reasons. First, AI scales in a way no human can: if you train an AI model on all the production music in the world, that model will be able to replace the demand for pretty much all of that music. No human can do this. Second, humans create within an implicit social contract – they know that people will learn from their work. This is priced in, and has been for hundreds of years. We don’t create work with the understanding that billion-dollar corporations will use it to build products that compete with us. This sits outside of the long-established social contract.
Do you think that legislators around the world are moving quickly enough to protect the rights of artists?
No. We need legislators to move faster. On current timetables, there is a serious risk that any solutions – such as enforcing existing copyright law, requiring companies to reveal their training data, etc. – will be too late, and these tools will be so widespread that it will be very hard to roll them back.
At Fairly Trained you provide a certification that signifies that a company trains their models on ‘data provided with the consent of its creators’. How do you acquire an accurate and transparent knowledge of the data each company is using?
They share their data with us confidentially.
For Fairly Trained to be successful it must earn people’s trust. What makes your organisation trustworthy?
We are a non-profit, and we have no financial backing from anyone on either side of this debate (or anyone at all, in fact). We have no hidden motives and no vested interests. I hope that makes us trustworthy.
If your ideal legislation existed, would a company like Fairly Trained be necessary?
No, Fairly Trained would not be necessary. I very much hope to be able to close it down one day!
To learn more about what you have read in this article you can visit the Fairly Trained website or Ed Newton-Rex’s website.
Dom Aversano is a British-American composer, percussionist, and writer. You can discover more of his work at the Liner Notes.
TouchDesigner Meetup – Interactivity
TouchDesigner Meetup – Interactivity
Getting Started with Stochastic Music in Max
Getting Started with Stochastic Music in Max
Music in the browser or app?
Dom Aversano
As The Bard famously put it, ‘The app, or the browser, that is the question.’
At some point, your inspirational idea for digital music will have to travel from the platonic realm of your thoughts, into either an app or browser. Unless you can luxuriate in doing both, this represents a stark choice. The most appropriate choice depends on weighing up the advantages and disadvantages of both. The graphic above is designed to help categorise what you are creating, thereby providing a better sense of its ideal home.
The most traditional category is recorded music, as it predates the proliferation and miniaturisation of personal computing. In the 20th Century, radio transformed music, and then television transformed it again. In this regard, Spotify and YouTube are quite traditional, as the former imitates radio while the latter mimics TV. This might help explain why Spotify is almost entirely an app, sitting in the background like a radio, and YouTube is most commonly used in the browser, fixing your gaze as if it were a TV. Whether a person is likely to be tethered to a computer or walking around with a phone, may help in deciding between browsers and apps.
Turning to generative music, a successful example of this in the browser is Generative FM, created by Alex Bainter, which hosts more than 50 generative music compositions that you can easily dip into. It is funded by donations, as well as an online course on designing generative systems. The compositions are interesting, varied, and engaging, but as a platform it’s easy to tune out of it. This might be because we are not in the habit of listening to music in the browser without a visual component. The sustainability of this method is also questionable since, despite there still being a good number of daily listeners, the project appears to have been somewhat abandoned, with the last composition having been uploaded in 2021.
Perhaps Generative FM was more suited to an app form, and there are many examples of projects that have chosen this medium. Artists such as Bjork, Brian Eno, and Jean-Michel Jarre have released music as apps. There are obvious benefits to this, such as the fact that an app feels more like a thing than a web page, as well as the commitment that comes from installing an app, especially one you have paid for — in the case of Brian Eno’s generative Reflection app, it comes at the not inconsiderable costs £29.99.
Yet, more than a decade since Bjork released her app Biophilia, the medium is still exceedingly niche and struggling to become established. Bjork has not released any apps since Biophilia, which would have been time-consuming and expensive to create. Despite Bjork’s app not having beckoned in a new digital era for music, this may be a case of a false start rather than a nonstarter. As app building gets easier and more people learn to program, there may be a breakthrough artist who creates a new form of digital music that captures people’s imaginations.
To turn the attention to music-making, and music programming in particular, there is a much clearer migratory pattern. Javascript has allowed programming language to work seamlessly in the browser. In graphical languages, this has led to P5JS superseding Processing. In music programming languages Strudel looks likely to supersede TidalCycles. Of the many ways in which having a programming language in the browser is helpful, one of the greatest is that it allows group workshops to run much more smoothly, removing the tedium and delays caused by faulty software. If you have not yet tried Strudel, it’s worth having a go, as you can get started with music-making in minutes by running and editing some of its patches.
The final category of AI — or large language models — is the hardest to evaluate. Since there is massive investment in this technology, most of the major companies are building their software for both browsers and apps. Given the gold rush mentality, there is a strong incentive to get people to open up a browser and start using the software as quickly as possible. Suno is an example of this, where you can listen to music produced with it instantly. If you sign it only takes a couple of clicks and a prompt to generate a song. However, given the huge running costs of training LLMs, this culture of openness will likely reduce in the coming years, as the companies seek to recuperate their backers’ money.
The question of whether to build something for the browser or an app is not a simple one. As technology offers us increasingly large numbers of possibilities, it becomes more difficult to choose the ideal one. However, the benefit of this huge array of options is that we have the potential to invent new ways of creating and presenting music that may not yet have been imagined, whether that’s in an app or browser.
Please feel free to share your thoughts and insights on creating for the browser or apps in the comment section below!
Dom Aversano is a British-American composer, percussionist, and writer. You can discover more of his work at Liner Notes.
Max Meetup – Chaos and Randomness
Max Meetup – Chaos and Randomness
Book Review: Supercollider for the Creative Musician
Dom Aversano
Several years ago a professor of electronic music at a London University advised me not to learn Supercollider as it was ‘too much of a headache’ and it would be better just to learn Max. I nevertheless took a weekend course, but not long after my enthusiasm for the language petered out. I did not have the time to devote to learning and was put off by Supercollider’s patchy documentation. It felt like a programming language for experienced programmers more than an approachable tool for musicians and composers. So instead I learned Pure Data, working with that until I reached a point where my ideas diverged from anything that resembled patching cords, at which point, I knew I needed to give Supercollider a second chance.
A lot had changed in the ensuing years, and not least of all with the emergence of Eli Fieldsteel’s excellent YouTube tutorials. Eli did for SuperCollider what Daniel Shiffman did for Processing/P5JS by making the language accessible and approachable to someone with no previous programming experience. Just read the comments for Eli’s videos and you’ll find glowing praise for their clarity and organisation. This might not come as a complete surprise as he is an associate professor of composition at the University of Illinois. In addition to his teaching abilities, Eli’s sound design and composition skills are right up there. His tutorial example code involves usable sounds, rather than simply abstract archetypes of various synthesis and sampling techniques. When I heard Eli was publishing a book I was excited to experience his teaching practice through a new medium, and curious to know how he would approach this.
The title of the book ‘SuperCollider for the Creative Musician: A Practical Guide’ does not give a great deal away, and is somewhat tautological. The book is divided into three sections: Fundamentals, Creative Techniques, and Large-Scale Projects.
The Fundamentals section is the best-written introduction to the language yet. The language is broken down into its elements and explained with clarity and precision making it perfectly suited for a beginner, or as a refresher for people who might not have used the language in a while. In a sense, this section represents the concise manual Supercollider has always lacked. For programmers with more experience, it might clarify the basics but not represent any real challenge or introduce new ideas.
The second section, Creative Techniques, is more advanced. Familiar topics like synthesis, sampling, and sequencing, are covered, as well as more neglected topics such as GUI design. There are plenty of diagrams, code examples, and helpful tips that anyone would benefit from to improve their sound design and programming skills. The code is clear, readable, and well-crafted, in a manner that encourages a structured and interactive form of learning and makes for a good reference book. At this point, the book could have dissembled into vagueness and structural incoherence, but it holds together sharply.
The final section, Large-Scale Projects, is the most esoteric and advanced. Its focus is project designs that are event-based, state-based, or live-coded. Here Eli steps into a more philosophical and compositional terrain, showcasing the possibilities that coding environments offer, such as non-linear and generative composition. This short and dense section covers the topics well, providing insights into Eli’s idiosyncratic approach to coding and composition.
Overall, it is an excellent book that every Supercollider should own. It is clearer and more focused than The Supercollider Book, which with multiple authours is fascinating, but makes it less suitable for a beginner. Eli’s book makes the language feel friendlier and more approachable. The ideal would be to own both, but given a choice, I would recommend Eli’s as the best standard introduction.
My one criticism — if it is a criticism at all — is that I was hoping for something more personal to the authour’s style and composing practice, whereas this is perhaps closer to a learning guide or highly-sophisticated manual. Given the aforementioned lack of this in the Supercollider community Eli has done the right thing to opt to plug this hole. However, I hope that this represents the first book in a series in which he delves deeper into Supercollider and his unique approach to composition and sound design.
Click here to order a copy of Supercollider for the Creative Musician: A Practical Guide
Dom Aversano is a British-American composer, percussionist, and writer. You can discover more of his work at the Liner Notes.
An interview with Blockhead creator Chris Penrose
Dom Aversano
Blockhead is an unusual sequencer with an unlikely beginning. In early 2020, as the pandemic struck, Chris Penrose was let go from his job in the graphics industry. After receiving a small settlement package, he combined this with his life savings and used it to develop a music sequencer that operated in a distinctively different manner from anything else available. In October 2023, three years after starting the project, he was working full-time on Blockhead, supporting the project through a Patreon page even though the software was still in alpha mode.
The sequencer has gained a cult following made up of fans as much as users, enthusiastic to approach music-making from a different angle. It is not hard to see why, as in Blockhead everything is easily malleable, interactive, and modulatable. The software works in a cascade-like manner, with automation, instruments, and effects at the top of the sequencer affecting those beneath them. These can be shifted, expanded, and contracted easily.
When I speak to Chris, I encounter someone honest and self-deprecating, all of which I imagine contributes to people’s trust in the project. After all, you don’t find many promotional videos that contain the line ‘Obviously, this is all bullshit’. There is something refreshingly DIY and brave about what he is doing, and I am curious to know more about what motivated him, so arranged to talk with Chris via Zoom to discuss what set him off on this path.
What led you to approach music sequencing from this angle? There must be some quite specific thinking behind it.
I always had this feeling that if you have a canvas and you’re painting, there’s an almost direct cognitive connection between whatever you intend in your mind for this piece of art and the actual actions that you’re performing. You can imagine a line going from the top right to the bottom left of the canvas and there is a connection between this action that you’re taking with a paintbrush pressing against the canvas, moving from top right down to left.
Do you think that your time in the graphics industry helped shape your thinking on music?
When it comes to taking the idea of painting on a canvas and bringing it into the digital world, I think programs like Photoshop have fared very well in maintaining that cognitive mapping between what’s going on in your mind and what’s happening in front of you in the user interface. It’s a pretty close mapping between what’s going on physically with painting on a canvas and what’s going on with the computer screen, keyboard and mouse.
How do you see this compared to audio software?
It doesn’t feel like anything similar is possible in the world of audio. With painting, you can represent the canvas with this two-dimensional grid of pixels that you’re manipulating. With audio, it’s more abstract, as it’s essentially a timeline from one point to another, and how that is represented on the screen never really maps with the mind. Blockhead is an attempt to get a little closer to the kind of cognitive mapping between computer and mind, which I don’t think has ever really existed in audio programs.
Do you think other people feel similarly to you? There’s a lot of enthusiasm for what you doing, which suggests you tapped into something that might have been felt by others.
I have a suspicion that people think about audio and sound in quite different ways. For many the way that digital audio software currently works is very close to the way that they think about sound, and that’s why it works so well for them. They would look at Blockhead and think, well, what’s the point? But I have a suspicion that there’s a whole other group of people who think about audio in a slightly different way and maybe don’t even realise as there has never been a piece of software that represents things this way.
What would you like to achieve with Blockhead? When would you consider it complete?
Part of the reason for Blockhead is completely selfish. I want to make music again but I don’t want to make electronic music because it pains me to use the existing software as I’ve lost patience with it. So I decided to make a piece of audio software that worked the way I wanted it. I don’t want to use Blockhead to make music right now because it’s not done and whenever I try to make music with Blockhead, I’m just like, no, this is not done. My brain fills with reasons why I need to be working on Blockhead rather than working with Blockhead. So the point of Blockhead is just for me to make music again.
Can you describe your approach to music?
The kind of music that I make tends to vary from the start. I rarely make music that is just layers of things. I like adding little moments in the middle of these pieces that are one-off moments. For instance, a half-second filter sweep in one part of the track. To do that in a traditional DAW, you need to add a filter plugin to the track. Then that filter plugin exists for the entire duration of the track, even if you’re just using it for one moment. It’s silly that it has to exist in bypass mode or 0% wet for the entire track, except in this little part where I want it. The same is true of synthesizers. Sometimes I want to write just one note from a synthesizer at one point in time in the track.
Is it possible for you to complete the software yourself?
At the current rate, it’s literally never going to be finished. The original goal with Patreon was to make enough money to pay rent and food. Now I’m in an awkward position where I’m no longer worrying about paying rent, but it’s nowhere near the point of hiring a second developer. So I guess my second goal with funding would be to make enough money to hire a second person. I think one extra developer on the project would make a huge difference.
It is hard not to admire what Chris is doing. It is a giant project, and to have reached the stage that it has with only one person working on it is impressive. Whether the project continues to grow, and whether he can hire other people remains to be seen, but it is a testament to the importance of imagination in software design. What is perhaps most attractive of all, is how it is one person’s clear and undiluted vision of what this software should be, which has resonated with so many people across the world.
If you would like to find out more about the Blockhead or support the project you can visit its Patreon Page.
Dom Aversano is a British-American composer, percussionist, and writer. You can discover more of his work at Liner Notes.
The persistence of misogyny in music technology
Dom Aversano
Last week the British House of Commons Women and Equalities Committee released their report into misogyny in music. It reached a stark and clear conclusion.
In this Report we have focused on improving protections and reporting mechanisms, and on necessary structural and legislative reforms. The main problem at the heart of the music industry is none of these; it is the behaviour of men—and it is almost always men.
Although the report is specific to the United Kingdom many of its findings could apply to other countries. One universal problem is the tendency for some men to view misogyny as a woman’s problem, even though men have greater power to eliminate it. For those of us working with music technology, this needs to be taken to heart, as the field comes out very badly in the report, especially concerning the gender imbalance for music producers, engineers, and songwriters.
In 2022, just 187 women and non-binary people were credited as either producer or engineer on the top 50 streamed tracks in 14 genres, compared to 3,781 men. Of all songwriters and composers who received a royalty in 2020 from their music being streamed, downloaded, broadcast, or performed, only one in six (16.7%) were women.
Music technology education does not fare better.
Participation rates show that music technology courses still show a stark gender imbalance, reflecting the lack of female representation in the production workforce, despite the technology’s increasing importance to modern musicians.
After reading this I was curious to know how Music Hackspace shaped up in this regard. While far from a comprehensive analysis, I decided to count the number of female and male teachers on the Music Hackspace Website and discovered 32 female teachers (35%) and 58 male teachers (65%). This is far from equal, but at least better than the ‘stark gender imbalance’ mentioned in the report. However, until it is equal, it is not good enough.
On a personal note, when writing this blog I try to keep bias and discrimination at the front of my mind, but I am aware I interview more men than women. This is more complicated than simply my intentions. When invited for an interview men have generally been more forthcoming than women and tend to be easier to locate and contact, especially given they often have more prominence within the musical world. It is not hard to imagine why women might be more reluctant to subject themselves to public attention, as they are criticised more than men and taken less seriously. In the government report, many female artists and managers were regularly mistaken for girlfriends.
The misogyny women experience in the public eye was grotesquely demonstrated recently when X/Twitter was flooded with deepfakes porn images of singer Taylor Swift just a few days before this year’s Grammy Awards. One does not have to be a music superstar to be subjected to such abuse. Last year in the Spanish town of Almendralejo more than 28 girls aged from 11 to 17 had AI-generated naked images created of them, with 11 local boys having been involved in the creation and circulation of the images, demonstrating that such threats now exist across all levels of society.
This is to say nothing of the wider patriarchal socio-political forces at work. This year the world will again be subjected to a presidential run by the convicted sex offender Donald Trump, who has bragged about sexually assaulting women and described his daughter as “voluptuous”. He is not alone, with social media-savvy men like Jordan Peterson and Andrew Tate promoting their misogynistic ideas to mass audiences of boys and men. These misogynistic ideas have been demonstrated to be algorithmically amplified by platforms such as TikTok such that Gen Z boys are more likely than Baby Boomers to believe that feminism is harmful.
Music should set a better example and act as a counter-cultural force against these movements. Historically, music has been a driver of social change, as one can create and share ideas with millions of people across the world rapidly. Women’s participation in this artistic activity should be equal to that of men, and for as long as it is not, it is men’s responsibility to help redress the power imbalance. In this respect, I will finish with the same quote from the House of Commons report, which lays out the root of the problem starkly.
The main problem at the heart of the music industry (…) is the behaviour of men—and it is almost always men.
Click here for the full report Misogyny in Music by the British House of Commons Women and Equaliti