a different flavor of audio programming
TRANSCRIPT
A different flavor of audio programming
We won’t talk about
● Real time programming● Signal processing● C++
A different flavor of audio programming
We won’t talk about
● Real time programming● Signal processing● C++
We will talk about
● Digital Audio Workstation (DAW) internals○ File formats○ Representation of time○ Automation points○ Audio rendering details
● Python
About me (@offlinemark)
● Previously security engineer○ Reversed binaries, file formats,
network protocols○ Lead developer for Manticore project:
analysis tool for reversing x86/ARM● Music producer & DJ
○ 10 years using DAWs○ REAPER, Garageband, Ableton Live, FL Studio
Disclaimer
Digital Audio Workstations (DAWs)
DAW Core
DAW Core
Reverb Distortion EQ Limiter
Plugins
DAW Core
Reverb Distortion EQ Limiter
Plugins
Audio Subsystem
Needs of DAW users
Audio
AudioPlugins
Needs of DAW users
Audio
“Workflow”
Plugins
???
Needs of DAW users
Workflow Problem Example:
Time Marker Export
DAW Core
Time Marker Export
DAW Core
Time Marker Export
Audio Subsystem
Time Markers
DAW Core
Time Marker Export
Audio Subsystem
Time Markers
DAW Core
.als, .flp
Project File
Time Markers
SaveProject
.als, .flp
DAW Core
Project File
Time Markers
Time Markers
.als, .flp
DAW Core
Project File
Time Markers
DAW CoreTime
Markers
LoadProject
.als, .flp
DAW Core
Third PartyTool
Project File
Time Markers
LoadProject
Time Markers
.als, .flp
DAW Core
Third PartyTool
Project File
Time Markers
Export
Time Markers
00:00 - A05:22 - B10:59 - C
markers.txt
DAW Core
???????????????????????????????
Third PartyTool
Project File
Time Markers
LoadProject
Time Markers
???
ReverseEngineering
Taking things apart to see how they work
● Fix● Learn● Extend
???????????????????????????????
��
???????????????????????????????
DAW Core
Third PartyTool
Project File
Time Markers
LoadProject???
DAW Core
Third PartyTool
Project File
Time Markers
LoadProject???
DAW Core
Third PartyTool
Project File
Time Markers
LoadProject???
DAW Core
Third PartyTool
Project File
Time Markers
LoadProject
Time Markers
Prior Art: Solving workflow problems
Live Enhancement Suite● By @DylanTallchief and @InvertedSilence
● Extends Live via desktop automation (AutoHotkey, Hammerspoon) ● New hotkeys, menus, features● http://enhancementsuite.me/
@bcrypt’s rekordbox scripts
● ableton-to-cues.py - convert between Ableton Live Warp Markers and Rekordbox cues
● https://github.com/diracdeltas/rekordbox-scripts
Time Marker Extraction
Our goal:● Third party tool● Analyze DAW project file● Extract time markers● Output as plain text
○ Marker text○ Marker time (mm:ss format)
The map:
1. Easy Mode2. Hard Mode3. Unreal Mode
Easy Mode
No tempo automation(Single static tempo)
Tempo automation: Dynamic BPM changes
Tempo automation: Dynamic BPM changes
Hard Mode
With tempo automation (naive impl, inaccurate)
(Dynamic tempo)
With tempo automation (fancy impl, accurate)
(DAW rendering engine details)
Unreal Mode
The map:
1. Easy Mode2. Hard Mode3. Unreal Mode4. Super Unreal Mode 😱
1. Easy Mode2. Hard Mode3. Unreal Mode4. Super Unreal Mode 😱
The map:
With nonlinear tempo automation
1. Easy Mode2. Hard Mode3. Unreal Mode4. Super Unreal Mode 😱
The map:
With nonlinear tempo automation
1. Easy Mode2. Hard Mode3. Unreal Mode4. Super Unreal Mode 😱
The map:
We will only coverlinear tempo automation today
Easy mode: No tempo automation
My first guess at the Marker structure:
struct Marker {int time; // millisecondsstring text;
};
Wrong! Times are stored in beat time
struct Marker {int beat_time; // beat timestring text;
};
DAWS store object times in beat time
Beat time is stable
Beat time is convertible to real timeusing the BPM (Beats per Minute)
Pseudocode
// 1. Parse markers and BPM// 2. Print outfor marker in markers { sec_time = marker.beat_time * (60/BPM) print_out(sec_time, marker.text)}
Pseudocode
// 1. Parse markers and BPM// 2. Print outfor marker in markers { sec_time = marker.beat_time * (60/BPM) print_out(sec_time, marker.text)}
Pseudocode
// 1. Parse markers and BPM// 2. Print outfor marker in markers { sec_time = marker.beat_time * (60/BPM) print_out(sec_time, marker.text)}
Easy ModeWhat we need:
● Markers (beat time, text)● Tempo (BPM)
Ableton Live Format (.als)
$ file demo.alsdemo.als: gzip compressed data...
Ableton Live Format (.als)
● Compressed XML document● Declarative object hierarchy● Human readable
(but undocumented)
Ableton Live Format (.als): Locators
Ableton Live Format (.als): Tempo (BPM)
FL Studio Format (.flp)
$ file demo.flpdemo.flp: data
FL Studio Format (.flp)
● Raw binary format● Not human readable,
“undocumented”
FLP Format Resources: Notes from devs (1999)
FLP Format Resources
● github.com/LMMS/lmms/ (10 years old)● github.com/andrewrk/PyDaw (10 years old)● github.com/monadgroup/FLParser
Helpful, but missing parts of the format.
FL Studio Format (.flp)
Header ● Header section: file signature, header len, ...● Data section: array of Event structures● Iterate and interpret events to construct state● Each event carries 1 piece of data (int/string) about
project*
*variable length data can contain multiple pieces of data via an array of structs
Events
FL Studio Format (.flp)
Header ● Event Structure○ ID field: 1 byte ID (0-255)○ Data field: Depends on ID
■ 0-63: int8/uint_8■ 64-127: int16/uint_16■ 128-191: int32/uint_32■ 192-255: variable length (additional size field)
Event 1
Event 2
Event 3
Event 4
...*variable length data can contain multiple pieces of data via an array of structs
FL Studio Format (.flp)
Header ● Up to 256 events● Relevant events:
○ MARKER_TIME (0x94) [uint32]: marker beat*○ MARKER_TEXT (0xcd) [UTF-16]: marker text○ TEMPO (0x9c) [uint32]: millibeat per minute
Event 1
Event 2
Event 3
Event 4
...*technically is a “pulse”, which is a beat subdivision
FL Studio Format (.flp)
Header ● Up to 256 events● Relevant events:
○ MARKER_TIME (0x94) [uint32]: marker beat*○ MARKER_TEXT (0xcd) [UTF-16]: marker text○ TEMPO (0x9c) [uint32]: millibeat per minute
Event 1
Event 2
Event 3
Event 4
...*technically is a “pulse”, which is a beat subdivision
Easy ModeWhat we need:
● ✅ Markers (beat time, text)● ✅ Tempo (BPM)
Pseudocode
// parse markers (beat, text) and BPMfor marker in markers { sec_time = marker.beat_time * (60/BPM) print_out(sec_time, marker.text)}
Hard mode: Tempo automation (naive)
Tempo automation
Handling tempo automation
Abstract Algorithm● Start at beginning of timeline● For each automation segment
○ Compute time elapsed during segment● Continue until we reach marker● Sum all elapsed time segments
How are automation points stored?
struct Point {int beat_time;int BPM_value;
};
Hard mode pseudocode
// parse markers (beat, text)// and tempo automation pointsfor marker in markers { sec_time = compute_sec_time(marker.beat_time, points) print_out(sec_time, marker.text)}
Hard mode pseudocode
// parse markers (beat, text)// and tempo automation pointsfor marker in markers { sec_time = compute_sec_time(marker.beat_time, points) print_out(sec_time, marker.text)}
Hard mode pseudocode
// parse markers (beat, text)// and tempo automation pointsfor marker in markers { sec_time = compute_sec_time(marker.beat_time, points) print_out(sec_time, marker.text)}
int compute_sec_time(marker_beat_time, points) { // Accumulation algorithm here}
Hard ModeWhat we need to do
● ✅ Markers (beat time, text)● 🤔 Automation points● 🤔 How to compute
elapsed time?
Computing time elapsed between points
Computing time elapsed = Physics I
🚗💨
Computing time elapsed = Physics I
Integrate to calculatedistance elapsed
Computing time elapsed = Physics I
✅
Can’t integrate the BPM curve directly
Can’t integrate the BPM curve directly
Can’t integrate the BPM curve directly
Must invert BPM to Minutes Per Beat (MPB)
✅
..or even Seconds Per Beat (SPB)
✅
Computing Time Elapsed
for a BPM automation
1. Construct BPM function2. Convert to SPB form3. Integrate SPB function
1. Construct BPM Function
2. Convert to SPB form
3. Integrate SPB function
✅
3. Integrate SPB function
3. Integrate SPB function
✅Time Elapsed: 2.77 seconds
Hard ModeWhat we need:
● ✅ Markers (beat time, text)● ✅ How to compute
elapsed time?● 🤔 Automation points
Ableton Live Format (.als): Tempo Automation Points
Ableton Live: Tempo Automation Points
Ableton Live: Tempo Automation Points
Ableton Live: Tempo Automation Points
Ableton Live: Tempo Automation Points
Ableton Live: Tempo Automation Points
Fl Studio: Tempo Automation Points
Fl Studio: Tempo Automation Points
Fl Studio: Tempo Automation Points
Channels
Playlist Items
Structures
struct Channel { int id; int dest_id; int param_id; Point[] points; // ...};
struct PlaylistItem { int channel_id; int start; int length; // ...};
Structures
struct Channel { int id; int dest_id; int param_id; Point[] points; // ...};
struct PlaylistItem { int channel_id; int start_beat; int length; // beat time // ...};
Approach● Parse all channels and playlist items
○ Events: CHANNEL_NEW (0x40), AUTOMATION_CHANNEL (0xe3), AUTOMATION_DATA (0xea), PLAYLIST_ITEMS (0xe9)
Approach● Filter channels for tempo automation channels
○ Channel dest_id = 0x4000 (Master Track), param_id = 0x5 (Tempo Parameter)
Approach● Filter playlist items for tempo automation playlist items
○ Using channel IDs of tempo automation channels
Approach● Render each playlist item’s points
○ Resolve global timeline positions of points
Approach● Render each playlist item’s points
○ Resolve global timeline positions of points
Channel 1:Point 1: Start + 2Point 2: Start + 10Point 3: Start + 14
Approach● Render each playlist item’s points
○ Resolve global timeline positions of points
Channel 1:Point 1: Start + 2Point 2: Start + 10Point 3: Start + 14
Item 1: Start: 10 Item 2: Start: 60
Approach● Render each playlist item’s points
○ Resolve global timeline positions of points
Channel 1:Point 1: Start + 2Point 2: Start + 10Point 3: Start + 14
Item 1: Start: 10Point 1: 12Point 2: 20Point 3: 24
Item 2: Start: 60Point 4: 62Point 5: 80Point 6: 84
Approach● Render each playlist item
○ Resolve global timeline positions of playlist item’s points
Approach● Render each playlist item
○ Resolve global timeline positions of playlist item’s points
Approach● Render each playlist item
○ Resolve global timeline positions of playlist item’s points
Approach● Merge playlist items
Hard ModeWhat we need:
● ✅ Markers (beat time, text)● ✅ Automation points● ✅ How to compute
elapsed time?
One little problem..
It should take 2.77 seconds
✅Time Elapsed: 2.77 seconds
What does the DAW say?
The DAW doesn’t match the theoretical result!
Lesson 1: Accuracy is relevant!
The quality of implementation will affect our accuracy.
Lesson 2: Accuracy defined by the DAW
At the end of the day, we must match the DAW.
Unreal mode: Tempo automation (fancy)
Enough physics, let’s talk audio rendering
DAWs can’t truly render audio at changing tempo
Approximate by sampling automation curve
Approximate by sampling automation curve
Requires new approach for calculating time elapsed!
1. Find BPMs at step points
2. Convert BPMs to SPB
3. Construct SPB stepwise function
4. Compute area under SPB stepwise function
Time Elapsed
One Key Parameter: Tempo Quantization Value
??
One Key Parameter: Curve Quantization
➔ DAW implementation detail➔ Not documented➔ Not stored in project file
Unreal ModeWhat we need:
● ✅ Markers (beat time, text)● ✅ Automation points● ✅ How to compute
elapsed time?● 🤔 Quantization value?
How to find tempoquantization?
???
How to find tempoquantization?
Give up and ask for help :(
How to find tempoquantization?
● ✅ Give up● Brute force
Brute forcing the quantization value
➔ Quantization is happening.➔ Will be a power of 2. (16th, 32rd, 64th note, ...)
➔ Limited possibilities in practice. (214th notes are impractical)
What we know:
1. Pick a known automation curve (60-120, 4 beats)
1. Pick a known automation curve (e.g. 60-120, 4 beats)
And remember what the DAW reported
2. Create experimental test harness
Standalone time calculation, parameterized on quantization
2. Create experimental test harness
Standalone time calculation, parameterized on quantization
Input: Start BPM, End BPM, # Beats, Quantization Value
2. Create experimental test harness
Input: Start BPM, End BPM, # Beats, Quantization Value
Output: Time Elapsed
2. Create experimental test harness
Output: Time Elapsed
Input: Start BPM, End BPM, # Beats, Quantization Value
2. Create experimental test harness
3. Try a bunch of values! See which matches the DAW.
3. Try a bunch of values! See which matches the DAW.
Ableton Live’s quantization value
FL Studio’s quantization value?
FL Studio’s quantization value?
How to find tempoquantization?
● ✅ Give up● ✅ Brute force● 👂Observe using
DAW
Hearing the quantization
Unreal ModeWhat we need:
● ✅ Markers (beat time, text)● ✅ Automation points● ✅ How to compute
elapsed time?● ✅ Quantization value
.als
Ableton Live Project Export
Time Markers
00:00 - A05:22 - B10:59 - C
markers.txt
.flp
FL Studio Project
Automation Points
Third Party Tool
⚙
.als
Ableton Live Project Export
Time Markers
00:00 - A05:22 - B10:59 - C
markers.txt
.flp
FL Studio Project
Automation Points
Third Party Tool
✅ Third party tool✅ Extracts time markers✅ Robust & high accuracy
⚙
● ✅ Third party tool● ✅ Analyze DAW project file● ✅ Extract time markers● ✅ Output as plain text
○ Marker text○ Marker time (mm:ss format)
Our goal:
Implementation & Evaluation
dawtool 🛠
● Python 3● Parsers for DAW formats
○ Ableton Live v8-10○ FL Studio v11-12, 20
● Time marker extraction● Theoretical & DAW modes● No dependencies*
Evaluation: Speed
Evaluation: Speed
Evaluation: Accuracy (Stress Tests)
DAW Max Observed Err.
Ableton Live
DAW Max Observed Err.
Ableton Live .001 s
Evaluation: Accuracy (Stress Tests)
1 ms error over 16 hour Live set 🤷♂
DAW Max Observed Error % Error
Live .001 s .000002
FL Studio
Evaluation: Accuracy (Stress Tests)
DAW Max Observed Error % Error
Live .001 s .000002
FL Studio .56 s
Evaluation: Accuracy (Stress Tests)
DAW Max Observed Error % Error
Live .001 s 0.000002
FL Studio .56 s 0.0029
Evaluation: Accuracy (Stress Tests)
Demo
Conclusion
Audio
“Workflow”
Plugins
Needs of DAW users
AudioPlugins
Project file manipulation
Needs of DAW users
AudioPlugins
Project file manipulation
Desktop Automation
Needs of DAW users
AudioPlugins
Project file manipulation
Time Marker Exporter
Needs of DAW users
AudioPlugins
Project file manipulation
What else?? 🤔
Needs of DAW users
Timestamps (https://timestamps.me)
People actually use it!
You are more powerful than you think.
Acknowledgements
Luke Van Seters
Ableton & Image Line
All Timestamps users
ADC ‘20 Organizers
Thank you.Mark Mossberg @offlinemarkgithub.com/offlinemark/dawtool
Timestampstimestamps.me