Playing a Symphony Orchestra with Python via real-time MIDI

Skip to the last section for audio examples!

Spitfire's BBCSO: A Virtual Symphony Orchestra, and Orchestral Composition in a DAW

In previous blog post Real-time MIDI in Python using sched and rtmidi we developed a setup where we can generate MIDI messages and schedule them to be sent at specific time to a DAW through a virtual MIDI port that'll play loaded VST instrument.

Let's built on top of that setup to play instruments of a Symphony Orchestra. Spitfire recently released BBC Symphony Orchestra to be the "new standard in orchestral sampling". They painstakingly took multiple recordings of each note on each instrument being played in various articulations (called "techniques") using multiple microphones with 3 different dynamics. Articulations can be legato, tremolo, staccato... Dynamics are piano, mezzoforte, forte. Here is a review.

The professional version is a sound engine that blends 600GB of samples according to the score you wrote in your DAW! But another stellar achievement of Spitfire is to release a simpler version called Discover which carries most essential aspects of a virtual symphony orchestra in only 200MB. It costs $49 and if you are a cheapskate like me you can apply for the free version and wait for 2 weeks before you are allowed to download it. For basic experimentation, enjoyment of playing with high quality orchestra samples, considering more serious investment in composition without requiring to own a second computer with a high-speed SSD just to load whole sample set, Discover is more than perfect.

We'll use it as the virtual instrument that'll play the scores we generate with Python. The way to use this VST is, in a DAW, to have a new track for each instrument and articulation that is going to be played in the piece that is going to composed. Loading a new instance of the VST for each track and choose the instrument and the articulation for that track. The usual way of composing pieces is to connect a MIDI keyboard to your computer and play individual voices one by one, probably on a metronome, then refine them in time with multiple takes. Or, enter notes and dynamics using regular keyboard and mouse using various editor UIs, most common one being a piano roll. Can search something like compose orchestral music in a daw to see how the standard process works.

People usually use templates, which are projects file with all or a subset of voices pre-loaded without any score in them. You can find Discover Templates that has all instruments and articulations loaded in a separate track, on their website.

Routing notes to individual voices

The DAW I'm using is Reaper. So, the Reaper template is my starting point. Usually, in manual composition (as opposed to "computer aided composition" we are aiming for) you enable recording performance data for one (or more) tracks and MIDI notes triggered on input ports that the DAW is listening to are routed to VST instruments connected to those tracks. Enabling recording in Reaper is called "arm"ing, which makes the red recording button emit light.

An older concept is MIDI channels which are useful if you want to store your music in a single MIDI file. Each channel can be routed to a different voice of a synth. However, with these advanced DAW project files, the easier mode of operation is to make each track to listen to all available MIDI ports and channels. You arm the track of the instrument you'll voice, play on your computer, record MIDI notes as your liking, un-arm it.

We want to be able to generate notes for all available instruments simultaneously, hence we'll arm all of them at the same time, therefore we have to choose a single, unique MIDI instrument for each instrument and so that we can send notes from Python into that MIDI channel without making each instrument to sound every note.

First problem we'll hit is that there are 16 MIDI channels. Which means using a single MIDI port, we cannot control and voice more than 16 instruments at the same time. Luckily, as we saw in the previous post, the utility software we use to connect Python to DAW using virtual ports allow us to create as many ports as we want. Discover has 33 instruments and 47 articulations. So, 3 virtual ports should be enough for us, providing 3 x 16 = 48 simultaneously controllable voices.

So, here is my loopMIDI virtual ports setup. (I chose 4 for the sake of it)

Then enable virtual ports in Reaper.

After that we are able to select any channel from any virtual port as the sole input of an armed track

Now, let's open the template project for BBC Discover and how it is structured, what instruments and articulations we have. It takes a little bit of time because it loads 47 instances of the VST.

As can be seen, the tracks are structured in an hierarchy. Root level is orchestral sections: Strings, Brass, Woodwinds, Percussion. Second level is instrument in those sections. Violins 1/2, Violas, Celli, Basses in Strings; Horns, Trumpets, Trombones, Tuba in Brass etc. Third level is the articulations: Long, Spiccato (short notes), Pizzicato (picking strings), Tremolo (bowing back and forth) for string instruments; long and stacatissimo for brass instruments etc.

Below is the VST UI where the preset for the articulation armed above is loaded. Each section have their own color: green strings, red brass, blue woodwinds, yellow percussion. Selected instrument is at the top "Brass: Horns a4". Selected articulation is in the middle right. I'm suspected another viable workflow is that, assuming each instrument is playing one note in a single articulation at a given time, to have one track per instrument and changing dynamics and articulation of that instrument on-the-fly while recording instead of having one track per articulation. Maybe in the future we'll switch that workflow, and it'll be easier on the computational resources, but for now I'm going with the more verbose workflow.

The template track (section, instrument, articulations) hierarchy structure is not exactly consistent with how the that hierarchy is structured in the VST when it comes to Percussions. But I won't obsess about the differences and won't edit the template to make them identical.

You are expected to arm the track with the instrument articulation you want and record a performance. What we want to do is arm all of them and set them to a unique (port, channel) combination.

Observe that the piano keys at the bottom are highlighted only for the range in which the instrument is playable. For example, horns can play notes from A1 to F5. These the notes for which the instruments are sampled.

In order to help with instrument selection for algorithmic composition, I thought it'll be a good idea to create a small dataset that catalogs available/playable instruments. Since it is not expected to change, easiest way is to have write this dataset in a tabular format in a file. I prepared dataset as in 3 spreadsheet.

First one "Enums" has unique values of each dimension (Section, Instrument etc.).

Second one has instrument ranges that I gathered manually.

Third one is a (not fully) denormalized table of "sounds". Where each sound that can possible be played is in a row where its Section, Instrument, Articulation, Range indicated. (Because different articulations of the same instrument usually have the same range, to prevent repetition, put ranges into their own table)

Dataset can be accessed on Google Drive BBC Symphony Discover Sounds Dataset. I downloaded them as CSVs and opened them in Notebook.

class Configuration(object):
    virtual_ports = ['loopMIDI Port 1 1', 'loopMIDI Port 2 2', 'loopMIDI Port 3 3', 'loopMIDI Port 4 4']
    enums_file = 'BBC Symphony Discover Sounds Dataset - Enums.csv'
    ranges_file = 'BBC Symphony Discover Sounds Dataset - Ranges.csv'
    sounds_file = 'BBC Symphony Discover Sounds Dataset - Sounds.csv'
    channels_per_port = 16

import csv
from enum import Enum

def to_enum_dict(items: List[str]):
    return {s.upper().replace(' ', '_'): s for s in items}

with open(Configuration.enums_file) as csv_file:
    reader = csv.reader(csv_file, delimiter=',')
    next(reader) # skip headers
    rows = list(reader)
    columns = zip(*rows)
    sections, instruments, articulations = [[item for item in col if item] for col in columns]
    Section = Enum('Section', to_enum_dict(sections))
    Instrument = Enum('Instrument', to_enum_dict(instruments))
    Articulation = Enum('Articulation', to_enum_dict(articulations))

with open(Configuration.ranges_file) as csv_file:
    reader = csv.reader(csv_file, delimiter=',')
    next(reader)
    rows = list(reader)
    RANGES = {r[0]: (r[1], r[2]) for r in rows}

Using Python dataclass define a Sound class.

from dataclasses import dataclass, field
from typing import List, Tuple
from music21.pitch import Pitch

@dataclass
class Sound(object):
    long_name: str
    section: Section
    instrument: Instrument
    articulation: Articulation
    short_name: str
    low: str
    high: str
    low_no: int = field(init=False)
    high_no: int = field(init=False)
    # will be set by router
    port: rtmidi.MidiOut = None
    channel: int = None

    def __post_init__(self):
        self.low_no = Pitch(self.low).midi
        self.high_no = Pitch(self.high).midi

SOUNDS = []
with open(Configuration.sounds_file) as csv_file:
    reader = csv.reader(csv_file, delimiter=',')
    next(reader)
    for row in reader:
        ix, section, instrument, articulation, short_name = row
        long_name = f'{section} - {instrument} - {articulation}'
        low, high = RANGES[short_name]
        snd = Sound(
            long_name, Section(section), Instrument(instrument), Articulation(articulation), short_name,
            low, high
        )
        SOUNDS.append(snd)

def open_ports():
    available_ports = rtmidi.MidiOut().get_ports()
    ports = []
    for vpname in Configuration.virtual_ports:
        if vpname not in available_ports:
            raise ValueError(f'Virtual Port {vpname} is not an available MIDI Out Port')
    for vpname in Configuration.virtual_ports:
        midi_out = rtmidi.MidiOut()
        ix = available_ports.index(vpname)
        midi_out.open_port(ix)
        ports.append(midi_out)
    return ports

OUTS = open_ports()

def make_routes(sounds, ports):
    cpp = Configuration.channels_per_port  # 16
    if len(ports) * cpp <= len(sounds):
        raise Exception(f'Not enough ports. There are {len(sounds)} sounds, and {len(ports)} ports.' 
                        'One port has only {cpp} channels.')
    for ix, sound in enumerate(sounds):
        sound.channel = ix % cpp
        sound.port = ports[ix // cpp]

make_routes(SOUNDS, OUTS)

Next step is very labor intensive. :-) After computing the routes, sound objects have the information of into which MIDI output port and MIDI channel they should send MIDI messages. Set those port and channels as the only inputs for respective tracks.

for snd in SOUNDS:
    print(snd.long_name, OUTS.index(snd.port) + 1, snd.channel + 1)

Example output.

Strings - Violins 1 - Long 1 1
Strings - Basses - Pizzicato 2 3
Brass - Horns a4 - Long 2 4
Brass - Tuba - Staccatissimo 2 13
Woodwinds - Clarinets a3 - Long 3 4
Woodwinds - Bassoons a3 - Staccatissimo 3 7
Percussion - Harp and Celeste - Harp Plucks 3 8
Percussion - Percussion - Untuned Percussion 3 11
Percussion - Tuned Percussion - Xylophone 3 14

One of the above lines means to select the track for Staccatissimo Bassoons in Brass and set it's input to virtual port 3 and channel 7.

Don't forget to save your project file as a template so that you don't have to repeat this input setting process each time you start a new algorithmic composition project. Also, you can see that this process is very custom. We are connecting a general programming language to a software that runs ~50 plugins via virtual ports. Even though the idea behind is simple, there is no special purpose software for this purpose.

Let's play one note. Choose an instrument.

Generate notes

Now comes the fun part. Let's start by asking one instrument to start and stop playing by sending MIDI messages. Import constants for on and off messages.

from rtmidi.midiconstants import NOTE_ON, NOTE_OFF

and choose a sound

horn_long = get_sound(SOUNDS, Section.BRASS, Instrument.HORNS_A4, Articulation.LONG)

print(horn_long.low_no, horn_long.high_no, OUTS.index(horn_long.port) + 1, horn_long.channel + 1)
# 33 77 2 4

See that this Horn "sound" can play notes from 33 to 77. It's routed to our 2nd virtual port, channel 4. Then in one cell run following command to make it start playing. Remember the bitwise OR operation to send the message to a single channel.

horn_long.port.send_message([NOTE_ON | horn_long.channel, 70, 100])

Enjoy the wake up call of mellow and majestic sounding horns. When you think that there is not enough oxygen left in the players' lungs, run the command in the next cell to let them breathe again.

horn_long.port.send_message([NOTE_OFF | horn_long.channel, 70, 0])

Note that the audio was not immediately cut off when we gave the off command. The instruments have a natural decay/release phase. Also there is reverb simulation which makes the sound linger a little bit longer.

In order to be able to schedule notes, i.e. play melodies, let's bring the scheduler logic from previous blog post:

import sched
import time

def schedule_note(scheduler, port, channel, midi_no, time, duration, volume):
    # print(port, midi_no, time, duration, volume)
    scheduler.enter(time + duration - 0.01, 1, port.send_message, argument=([NOTE_OFF | channel, midi_no, 0],))
    scheduler.enter(time, 10, port.send_message, argument=([NOTE_ON | channel, midi_no, volume],))

and with each instrument and articulation in the symphonic library, play the note in the middle of its range.

s = sched.scheduler(time.time, time.sleep)
for i, snd in enumerate(SOUNDS):
    mid_no = (snd.low_no + snd.high_no) // 2
    schedule_note(s, snd.port, snd.channel, mid_no, i*0.2, 0.19, 100)
s.run();

Can't call it "music to my ears" but finally we hear our complex setup working!

Generate random melodies

A slightly more complex algorithm to generate orchestral phrases is

• choose a random instrument and articulation (uniform distribution)
• choose a random note from the available range of that instrument (again uniform distribution)

Visualize Ranges

def plot_instrument_ranges(sounds):
    inst_info = {(snd.short_name, snd.low_no, snd.high_no, snd.section): None for snd in sounds}.keys()
    color_map = {Section.PERCUSSION: 'orange', 
                 Section.WOODWINDS: 'green', 
                 Section.BRASS: 'red', 
                 Section.STRINGS: 'blue'}
    names, lows, highs, sections = zip(*inst_info)
    fig, ax = plt.subplots(figsize=(6, 6))
    ax.barh(
        y=list(range(len(names))), 
        left=lows,
        width=[high - low for (low, high) in zip(lows, highs)], 
        tick_label=names,
        color=[color_map[sec] for sec in sections]
    )
    # grid at Cs and Gs
    xticks = sorted(list(range(24, 111, 12)) + list(range(19, 111, 12)))
    ax.set_xticks(xticks)
    ax.set_xticklabels([Pitch(no) for no in xticks], rotation=0)
    ax.grid(axis='x', which='major')
    ax.set_title('Instrument Ranges')
    plt.tight_layout()
    plt.show()

plot_instrument_ranges(SOUNDS)

Basically the algorithm is to choose a note (x-coordinate) and then choose an instrument (y-coordinate) of which range bar includes that note with a random articulation.

import random

def random_sound_note(sounds):
    snd = random.choice(sounds)
    no = random.randint(snd.low_no, snd.high_no)  # inclusive
    return snd, no

Now, each time we run following cell in the Notebook, we'll generate another orchestral phrase.

s = sched.scheduler(time.time, time.sleep)
for i in range(12):
    snd, mid_no = random_sound_note(SOUNDS)
    schedule_note(s, snd.port, snd.channel, mid_no, i*0.15, 0.14, 100)
s.run();

I'm definitely not going to claim that these random phrases sound like late 20. century compositions. :-P

Playing with this makes me feel like Jackie Chan from Twin Dragon, where he is the twin of an orchestra conductor with whom he accidentally switches places and he controls the orchestra by waving his hands around randomly. :-)

In a world where software was more portable, I could have placed a button here and you could generate a new musical phrase and let it be played by BBC Symphony Orchestra and listen to it in real-time. But unfortunately our world is not that world, yet.