Video presentation I made for the 2024 “Explainable AI for the Arts” (XAIxArts) Workshop, part of the ACM Creativity and Cognition Conference 2024.
A lot of these points I’ve discussed elsewhere (see playlist below), but this quickie presentation brings together these ideas, focusing on the aesthetic potential of this approach.
Check out the complete playlist for more hands-on creation of neurons and neural networks:
Building a matrix mixer in Reaktor 6 primary, and then exploring its possibilities for sound design.
I’ve been a little obsessed with matrix mixers lately, because they feed my desire for unique sound design applications (and feedback). A matrix mixer is a combiner module that can route multiple inputs to multiple outputs, often allowing you to adjust how much of each input signal goes to each output. While sophisticated, they’re pretty easy to build in Reaktor or other software, and can maybe be useful for some next-level synthesis applications.
0:00 Intro / Why Matrix Mixers?
1:18 Starting the Build, Simple Sine Oscillator
2:50 Matrix Mixer Macro
3:55 Visible Ports for Panel Patching
5:37 First Mixer of the Matrix
9:46 Duplicating It for the Matrix
10:48 Matrix Mixer Basics
12:05 Delay and Feedback
15:29 Adding a Second Delay
18:44 END OF LESSON. Unless…
19:59 Adding Ring Modulation
21:43 Building a Complex Patch
24:45 Other Examples of Implementation
25:58 Final Thoughts, Next Steps
Beginning Reaktor 6 Tutorials:
Adding some subtle dimension to your synthesized instruments with early reflections.
Synthesized instruments, unlike recorded sounds, have never existed in the acoustic world. This means that these synthesized sounds are 100% direct signal. To add some subtle dimension to these synthesized sounds, then, we can craft some “early reflections” on these tracks.
Here, I demonstrate this concept using ChromaVerb in Logic Pro X.
0:00 Introduction
0:55 Understanding Reverb and Early Reflections
2:18 Creating Early Reflections
3:18 Project Setup
4:26 Early Reflections Aux Track
5:15 ABing on the Synth Track
6:19 Why Separate these from the Main Reverb?
6:40 ABing on the Whole Arrangement
Doing some “samplecrushing” (downsampling) in Pure Data Vanilla to create dynamic aliasing artifacts.
0:00 Setting up [samphold~]
0:28 Simple downsampling and aliasing
0:55 Building a sequencer
2:33 Making the samplecrush dynamic
3:13 Making it stereo
3:50 Trying different timings and ranges
More Pd patching from scratch here:
Combining human input from a joystick with a two-neuron artificial neural network for chaotic interactive music.
This Eurorack joystick is going into a simple neural network to control multiple dimensions of the timbre of this synth voice. Joystick dimensions X, Y, and Z go into different inputs of the Nonlinear Circuits Dual Neuron, and are mixed together and transformed by a nonlinearity (more here). In addition to the output controlling the waveform and filter cutoff of the synth, the outputs of each neuron is fed back into the other, creating a chaotic artificial organism with which to improvise.
Affiliate links for modules in this patch (though you really don’t need them; you can probably work this out with the gear or software that you currently have):
–Doepfer A-174-4 3D Joystick (Perfect Circuit)
–NLC Dual Neuron (Reverb)
–Noise Engineering Ataraxic Translatron (Reverb)
–Hikari Ping Filter (Perfect Circuit)
–Noise Engineering Sinclastic Empulatrix (Reverb)
–Arturia DrumBrute Impact (Perfect Circuit)
–Korg SQ-1 (Perfect Circuit)
More Music with Artificial Neurons:
An overview of MIDI System messages and how they can support MIDI programming and synchronization in your studio.
I ran away from an explanation of system messages in my previous video on MIDI Messages, instead focusing entirely on channel messages. In this video, though, I’m back to talk about System Exclusive Messages, System Common Messages, and System Realtime Messages, and how you can implement them for additional musical control.
0:00 Introduction
0:22 Quick Review of bits and bytes
0:57 Channel vs. System Messages
1:59 Categories of System Messages
2:36 System Exclusive (SysEx) Messages
4:50 System Common Messages
5:08 Song Select, Song Position Pointer
6:38 MIDI Time Code
7:31 Time Code Quarter Frame Message
9:10 Tune Request Message
9:58 System Real Time Messages
10:41 Active Sensing
11:25 Reset Message
11:56 MIDI Clock, Start, Continue, & Stop
12:39 MIDI Sync Demo in Max
13:06 MIDI Sync Demo in Logic Pro X
13:26 Wrap-up
MIDI Protocol 1: Bits, Bytes, and Binary
MIDI Protocol 2: MIDI Messages
Building a simple artificial neural network in Max/MSP for nonlinear, chaotic control of data-driven instruments.
I’ve talked before about data-driven instruments, and I’ve talked before about artificial neurons and artificial neural networks, so here I combine the ideas to use a simple neural network to give some chaotic character to incoming data from a mouse and joystick before converting into into MIDI music. The ANN (Artificial Neural Network) reinterprets the data in way that isn’t random, but also isn’t linear, perhaps giving some interesting “organic” sophistication to our data-driven instrument.
In this video, I work entirely with musical control in MIDI, but these ideas could also apply to OSC or directly to any musical characteristics (like cutoff frequency of a filter, granular density, etc.).
0:00 Intro
1:43 [mousestate] for Data Input
2:58 Mapping a linear data-driven instrument
7:19 Making our Artificial Neuron
15:27 Simple ANN
20:06 Adding Feedback
22:23 Closing Thoughts, Next Steps
More Max/MSP Videos:
More Artificial Neurons and Neural Networks:
Turning a single cycle of a recorded sample into a wavetable for Kyma oscillators.
When composing music with samples, it’s worthwhile to explore all of the musical opportunities in that sample–reversing it, timestretching it, granulating it, etc. Along those same lines, you can take a wavetable fro a sample and use it in your oscillators, so, instead of using the usual sawtooth, square, or sine waves, you create an oscillator that has a timbral connection to the sampled material.
Here, I show how to take take two vowel sounds from a vocal sample–an “ah” and an “oh”–and cycle them in a Kyma oscillator, creating unique timbres that blend with the original sample and its processing.
0:00 Intro / Why?
0:41 Finding a Single Cycle
3:14 Changing Duration to 4096 Samples
4:16 Cycling the Wavetable in an Oscillator
6:33 Making a Different Oscillator Wavetable
9:21 Implementation Example: Chords
11:49 Adding Vibrato
14:08 SampleCloud Plus Chords
More Symbolic Sound Kyma videos:
An improvised duet(?) with a AI agent trained on the “Embodied Musicking Dataset. “
In this performance, Python listens to live audio input from the bass, and, based on models trained with the dataset, sends out data to Unity3D and Kyma. Unity3D creates the visuals (the firework), and Kyma processes the audio from the bass.
First, though, the dataset used for training was collected from several pianists in the US and UK. As pianists played, we recorded multiple aspects of their performance: audio, video of their hands, EEG, skeletal data, and galvanic skin response. After playing, pianists listened to their own performance and were asked to record their state of “flow” over the course of the performance. All of these different dimensions of data, then, were associated over time, and so neural networks can be trained on these different dimensions to make associations.
This demonstration uses the trained models from Craig Vear’s Jess+ project to generate X&Y data (from the skeletal data), and “flow”, from the amplitude of the input. These XY coordinates, “flow”, and amplitude are sent out from Python as OSC Data, which is received by both Unity3D (for visuals) and Kyma (for audio).
In Unity, the XY data moves the “firework” around the screen. Flow data affects its color, and amplitude affects its size. Audio in Kyma is a bit more sophisticated, but X position is left/right pan, and the flow data affects the delay, reverb, and live granulation.
As you can see, amplitude to XY mapping is limited, with the firework moving along a kind of diagonal. Possible next steps would be to extract more features of the audio (e.g. pitch, spectral complexity, or delta values), and train with those.
Applying this data trained on pianists to a bass performance (in a different genre) does not have the same goals music-generation AI such as MusicGen or MusicLM. Instead of automatically generating music, the AI becomes a partner in performance. Sometimes unpredictable, but not random, since its behavior is based on rules.