+ - 0:00:00
Notes for current slide

Sound check. Reset clock before starting.

Notes for next slide

Making Music With Code

1 / 107

Sound check. Reset clock before starting.

What is This?

2 / 107

What Is This

3 / 107

What Is This

  • A spiritual sequel of a demo/talk I gave at Brooklyn.js back in 2018 about making hip hop with JavaScript.
4 / 107

What Is This

  • A spiritual sequel of a demo/talk I gave at Brooklyn.js back in 2018 about making hip hop with JavaScript.

  • An overview of algorithmic composition with some ML approaches covered at the end.

5 / 107

What Is This

  • A spiritual sequel of a demo/talk I gave at Brooklyn.js back in 2018 about making hip hop with JavaScript.

  • An overview of algorithmic composition with some ML approaches covered at the end.

  • A collection of data viz, tech demos & experiments using WebAudio API and various JavaScript/Python tools.

6 / 107

Who Am I?

7 / 107

Who Am I?

8 / 107

Who Am I?

9 / 107

Who Am I?

10 / 107

Who Am I?

  • I'm omar delarosa. 👋

  • I'm a Senior Engineer at Spotify. 🎶

  • I play music in my spare time. 🎸 🎹

11 / 107

Who Am I?

  • I'm omar delarosa. 👋

  • I'm a Senior Engineer at Spotify. 🎶

  • I play music in my spare time. 🎸 🎹

  • I studied (almost) a minor's-worth of music theory courses. 👶🎼

12 / 107

Who Am I?

  • I'm omar delarosa. 👋

  • I'm a Senior Engineer at Spotify. 🎶

  • I play music in my spare time. 🎸 🎹

  • I studied (almost) a minor's-worth of music theory courses. 👶🎼

  • I studied a master's-worth of computer science courses. 🤖🎓

13 / 107

Who Am I?

  • I'm omar delarosa. 👋

  • I'm a Senior Engineer at Spotify. 🎶

  • I play music in my spare time. 🎸 🎹

  • I studied (almost) a minor's-worth of music theory courses. 👶🎼

  • I studied a master's-worth of computer science courses. 🤖🎓

  • I put music on Instagram and YouTube sometimes.

14 / 107

Why This?

15 / 107

Why This?

16 / 107

Why This?

  • Maybe you know how to code 💻
17 / 107

Why This?

  • Maybe you know how to code 💻

  • ... but you don't play any instruments

18 / 107

Why This?

  • Maybe you know how to code 💻

  • ... but you don't play any instruments

  • ... and you want to make music 🎸

19 / 107

Why This?

  • Maybe you know how to code 💻

  • ... but you don't play any instruments

  • ... and you want to make music 🎸

  • This offers some ideas 🤔

20 / 107

Agenda

  • Basics 👶
21 / 107

Agenda

  • Basics 👶

    • Music Theory 🎶
22 / 107

Agenda

  • Basics 👶

    • Music Theory 🎶

    • Music Graphs 🕸

23 / 107

Agenda

  • Basics 👶

    • Music Theory 🎶

    • Music Graphs 🕸

    • Markov Chains ⛓

24 / 107

Agenda

  • Basics 👶

    • Music Theory 🎶

    • Music Graphs 🕸

    • Markov Chains ⛓

  • Advanced 👴
25 / 107

Agenda

  • Basics 👶

    • Music Theory 🎶

    • Music Graphs 🕸

    • Markov Chains ⛓

  • Advanced 👴

    • Vectorizing Music ♾
26 / 107

Agenda

  • Basics 👶

    • Music Theory 🎶

    • Music Graphs 🕸

    • Markov Chains ⛓

  • Advanced 👴

    • Vectorizing Music ♾

    • Machine Learning and Music 🤖

27 / 107

What is Music? 🤔

28 / 107

Sound Waves

29 / 107

Individual Frequencies Are Perceived as Tones

30 / 107

Random Tone Frequencies

31 / 107

Special Frequencies

32 / 107

Special Frequencies

  • Frequences like 440hz are named notes such as A4 (more info)
33 / 107

Special Frequencies

  • Frequences like 440hz are named notes such as A4 (more info)

  • Exact tunings can vary by instrument and styles

34 / 107

Special Frequencies

35 / 107

Special Frequencies

  • Definitions:
    • a note is named tone
    • a scale is ordered set of notes
36 / 107

Random Named Notes from the Same Scale

37 / 107

Harmony

38 / 107

Harmony

39 / 107

Harmony

  • Relationships between sound waves
40 / 107

Harmony

  • Relationships between sound waves

  • Often occurs when multiple waves oscillate at whole number ratios with each other

  • Definitions:

41 / 107

Harmony

  • Relationships between sound waves

  • Often occurs when multiple waves oscillate at whole number ratios with each other

  • Definitions:

    • melody - a sequence of tones over time
    • chord - tones at the same time, named after the component notes
    • chord progression - a sequence of chords over time:
42 / 107

Rhythm

43 / 107

Rhythm

  • duration - how long a tone lasts

  • beat - a single unit of rhythm

  • measure - a regularly spaced group of beats

44 / 107

Rhythm & Fractions

  • Durations are all described as fractions of a measure
45 / 107

Rhythm & Fractions

  • Durations are all described as fractions of a measure

  • 1/4 Note

46 / 107

Rhythm & Fractions

  • Durations are all described as fractions of a measure

  • 1/4 Note

  • 1/8 Note

47 / 107

Rhythm & Fractions

  • Durations are all described as fractions of a measure

  • 1/4 Note

  • 1/8 Note

  • 1/16 Note

48 / 107

Fun With Fractions

  • Not all are multiples of 2.
49 / 107

Fun With Fractions

  • Not all are multiples of 2.

  • Some interesting things happen when you mix up durations where the denominator of the fraction is a multiple of 3.

50 / 107

Fun With Fractions

  • Not all are multiples of 2.

  • Some interesting things happen when you mix up durations where the denominator of the fraction is a multiple of 3.

  • This is common in generes such as hip hop.
51 / 107

Harmony & Rhythm = Pitch & Time

Chopin's Etudes, Opus 10, No.1 as a pianoroll plot

52 / 107

Graphs

53 / 107

Music & Graphs

Music sequences can be thought of as directed walks along the edges of a graph of the state-space in which each discrete* state is a node.

Each state (Sn) can be a single note or a beat or any music element in a sequence of musical events.

*Assume each state in music is discrete. Further discussions of microtonality and certain continuous musical spaces are out of scope in this section.

54 / 107

Music & Finite State Machines

The entire action space of a sequence of music can thus be described using a finite state machine.

55 / 107

Markov Chain

The most straightforward way to produce algorithmic compositions.

"a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event"

from Wikipedia

56 / 107

Markov

Andrey Markov, namesake of Markov chains and more.

57 / 107

Note Markov Chain

58 / 107

Avoid talking over synths

Chord Markov Chain

59 / 107

Avoid talking over synths

Beat Markov Chain

60 / 107

Drake Markov Chain

61 / 107

Markov Chains (With Benefits)

62 / 107

Markov Chains (With Benefits)

  • Markov chain transition matrices can be learned from a corpus text (or a MIDI file)
63 / 107

Markov Chains (With Benefits)

  • Markov chain transition matrices can be learned from a corpus text (or a MIDI file)

  • Data can be represented and stored easily as structured data formats such as JSON

64 / 107

Markov Chains (With Benefits)

  • Markov chain transition matrices can be learned from a corpus text (or a MIDI file)

  • Data can be represented and stored easily as structured data formats such as JSON

  • Can extend to more complex, multi-state transition matrices, but this is outside the scope of this prez.

65 / 107

Music & Code

66 / 107

Rhythm as Code

Beat Grids

  • Centered around 1/16th note ticks

  • Can be difficult to "escape the grid" with durations < 1/16.

  • Also tough to use subdivisions that are multiples of 3.

67 / 107

Rhythm and Computation

Beat Grids as Code

// 16-element arrays can represent rhythm patterns, but are tough to read.
const kicks = [1,0,1,0, 0,0,1,0, 0,0,1,0, 0,0,1,0];
const snares = [0,0,0,0, 1,0,0,0, 0,0,0,0, 1,0,0,0];
const hats = [1,1,1,1, 1,1,1,1, 1,1,1,1, 1,1,1,1];
const cowbell = [0,0,0,0, 0,0,0,0, 0,0,0,0, 1,0,1,0];
68 / 107

Rhythm Notation

Beat Grids as "Words"

// Easier to read, CPU-trivial preprocessing
const kicks = fmt('1010 0010 0010 0010');
const snares = fmt('0000 1000 0000 1000');
const hats = fmt('1111 1111 1111 1111');
const cowbell = fmt('0000 0000 0000 01010');
69 / 107

Rhythm Notation

Beat Grids as Lists of Words (a Language?)

// A list of patterns
const kick_patterns = [
fmt("1010 0010 0010 0010"),
fmt("1001 0001 0101 0010"),
fmt("1000 0101 0100 0010"),
fmt("1000 0010 0000 0100"),
];
70 / 107

Generative Beats

We could make two pattern sets

const kick_patterns = [
fmt("1010 0010 0010 0010"),
fmt("1001 0001 0101 0010"),
fmt("1000 0101 0100 0010"),
fmt("1000 0010 0000 0100"),
];
const snare_patterns = [
fmt("0000 1000 0000 1000"),
fmt("0010 1000 0000 1010"),
fmt("0000 1000 0010 1000"),
];
71 / 107

Generative Beats

And randomly combine them

const kicks_sequence = [
..._.sample(kick_patterns),
..._.sample(kick_patterns),
..._.sample(kick_patterns),
..._.sample(kick_patterns),
];
const snare_sequence = [
..._.sample(snare_patterns),
..._.sample(snare_patterns),
..._.sample(snare_patterns),
..._.sample(snare_patterns),
];
playParallel(kicks_sequence, snare_sequence);
72 / 107

Generative Beats

Simple Markov Chain Implementation

class MarkovChain {
constructor(obj = {}, states = [], initialState = 0) {
this.graph = { ...obj };
this.states = [...states];
this.currentState = initialState;
}
set() {
const newState = this.sample(this.graph[this.currentState]);
this.currentState = newState;
}
next() {
this.set();
return this.states[this.currentState];
}
sample(list) {
return list[Math.floor(list.length * Math.random())];
}
}
73 / 107

Markov Chain of Notes

  • Using an adjacency list instead of matrix (for readability, simplicity)
const NOTES = ["C", "D", "E", "F", "G", "A", "B"];
const G = {
// Repeated notes represent higher probabilities
0: [1, 1, 0, 3, 4, 5, 6], // 0 -> 1 is 2/7, the rest 1/7
1: [0, 0, 2, 3], // 1 -> 0 is 1/2 the others 1/4
2: [1, 3, 4],
3: [4], // 3 -> 4 means state 4 always follows 3 or 1/1 probability
4: [5],
5: [5, 4, 1, 0],
6: [2, 2, 2, 3, 3],
};
const mc = new MarkovChain(G, NOTES);
74 / 107

Markov Chain of Chords

const CHORDS = [
"C maj", "D min", "E min", "F maj", "G maj", "A min", "B dim"];
// Favors I <-> IV, V -> I cadences
const G = {
0: [3, 3, 3, 5],
1: [2, 5],
2: [3],
3: [4, 4, 4, 1, 1],
4: [0, 0, 0, 5],
5: [1, 6],
6: [4],
};
const mc = new MarkovChain(G, CHORDS);
75 / 107

Markov Chain of Rhythm Patterns

// Tuples of duration & number of beats
const HATS = [
[16, 4],
[12, 3],
[24, 6],
[32, 4],
[48, 6],
[64, 8],
];
// Favors steady 1/16 notes -- common in hip hop
const G = {
0: [0, 0, 0, 0, 0, 0, 1, 2, 3, 4], // 0 -> 0 has 3/5 odds
1: [0, 0, 0, 3],
2: [0, 0, 0, 3],
3: [2, 5],
4: [2, 3, 4, 1],
5: [3, 2, 4, 2, 2],
};
76 / 107

And So...

  • Music is a graph.
77 / 107

And So...

  • Music is a graph.

  • Music sequences are walks along the edges of those graphs.

78 / 107

And So...

  • Music is a graph.

  • Music sequences are walks along the edges of those graphs.

  • Markov chains are cool.

79 / 107

Advanced Topics

80 / 107

Remember Pitch & Time?

Chopin's Etudes, Opus 10, No.1 as a pianoroll plot

81 / 107

Let's Turn That on Its Side


82 / 107

Discrete Note Spaces

83 / 107

1-D Row Vector Note States

84 / 107

Stacked Row Vectors

Let s = a given state, t = a point in time and i = a note in some scale.

2-D Matrices Representing Pitch (Columns) and Time (Rows)

85 / 107

Now Let's Get Creative

86 / 107

Hypercube Music

87 / 107

Walking a Hypercube Graph

When scaled up to an 8-D hypercube this can easily represent any diatonic scale such as C Major or B minor.

88 / 107

Hypercube Music Example

And arpeggiator sequence of nearest neighbor "chord nodes" on an 8-D hypercube:

89 / 107

Cellular Automata Music

90 / 107

Cellular Automata Music

Generate row vectors using CA update rules:


Stack them to form music state matrices:

91 / 107

Cellular Automata Music

Sample notes from the state matrices to make melodies or chords:

92 / 107

Cellular Automata Music

Example 1: CA Piano

93 / 107

Cellular Automata Music

(see Wolfram Tones for more elaborate examples)

94 / 107

Machine Learning Music

95 / 107

Chord2Vec

Using a data set of MIDI files, transform notes into chord-name strings and then vectors.

(see word2vec for more information on how this works.)

96 / 107

Chord2Vec

This results in a vector space made of each chord in your dataset

(Each point in this space can be thought of as a node in a graph)

97 / 107

Chord2Vec

Each chord has some number K nearest neighbors in this space.

98 / 107

Chord2Vec

Chords are expanded from single note inputs by selecting a nearest neighbor (randomly or by whatever other means)

99 / 107

LLM Music via Suno Prompt

Prompt: Make a chill track in the style of lofi hip hop

100 / 107

And So...

  • Beyond Markov Chains there are many other ways to compose algorithmically.
101 / 107

And So...

  • Beyond Markov Chains there are many other ways to compose algorithmically.

  • You can use cellular automata

102 / 107

And So...

  • Beyond Markov Chains there are many other ways to compose algorithmically.

  • You can use cellular automata

  • Hypercubes

103 / 107

And So...

  • Beyond Markov Chains there are many other ways to compose algorithmically.

  • You can use cellular automata

  • Hypercubes

  • NLP tools

104 / 107

And So...

  • Beyond Markov Chains there are many other ways to compose algorithmically.

  • You can use cellular automata

  • Hypercubes

  • NLP tools

  • LLMs and beyond

105 / 107

Thanks!

107 / 107

What is This?

2 / 107
Paused

Help

Keyboard shortcuts

, , Pg Up, k Go to previous slide
, , Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow