Orchestrating the Web Audio API

by Srikumar K. Subramanian JSFoo, Bangalore

18 Sep 2015

Licensed under Creative Commons CC-BY-SA 3.0

Agenda

Agenda• A personal history of noise making and computers

Agenda• A personal history of noise making and computers

• Architecture of the Web Audio API - Dynamic signal flow graphs

Agenda• A personal history of noise making and computers

• Architecture of the Web Audio API - Dynamic signal flow graphs

• Taming the dynamism

Agenda• A personal history of noise making and computers

• Architecture of the Web Audio API - Dynamic signal flow graphs

• Taming the dynamism

• Just-in-time scheduling

Agenda• A personal history of noise making and computers

• Architecture of the Web Audio API - Dynamic signal flow graphs

• Taming the dynamism

• Just-in-time scheduling

• … and how it’s going to get better soon

Agenda• A personal history of noise making and computers

• Architecture of the Web Audio API - Dynamic signal flow graphs

• Taming the dynamism

• Just-in-time scheduling

• … and how it’s going to get better soon

• … with demos, of course.

In the beginning _

Seymour Papert

“Logo” (1967)

BEEP duration, pitch

SOUND channel, volume, pitch, duration

PLAY "cdefgab"

What about today?

Before the Web Audio API, . . .

TOUGH LUCK!

BEEP using the Web Audio API

x gain Filter Out

Examples

CSound (since 1986)

Max (since 1986)

Pure Data (since 1990)

SuperCollider (since 1996)

SuperCollider (since 1996)

Chuck (since 2003)

The Web Audio API changes everything

DEMO

The “Rhythm Engine”

Revisiting beep() …

Now you see me Now you don’t

Problem 1

x gain Filter

Out

beep(330,2.0) sets this up at first

x gain Filter

Out

beep(330,2.0) sets this up at first

About 2.0 seconds later …

x gain Filter

Out

Gain

model.beep(330,2.0)

x gain Filter

Out

Gain

model.beep(330,2.0)

Gain

About 2.0 seconds later … we can beep again.

Changing our minds - a.k.a. interactivity

Problem 2

setTimeout() to the rescue?

setTimeout() to the rescue?

NOT!

Just-in-time schedulingusing the Web Audio API and

requestAnimationFrame()

… together.

requestAnimationFrame() callbacks

scheduling intervals with native node types

time

time

requestAnimationFrame() callbacks

longer scheduling intervals when using ScriptProcessor

Steller gives you …

• A GraphNode abstraction that is stable for model construction.

• A sample-accurate just-in-time Scheduler with multi-tracking and tempo control.

• Decoupling of specification of interactive models from their performance.

GraphNode

function GraphNode(node, inputs, outputs) {...}

Turns any object into a node that can participate in the

audio graph.

Scheduler

function (sched, clock, next) { // ... do something next(sched, clock, sched.stop); // Go to the next one (optional). }

Scheduler

delay, track, spawn, sync, gate, rate, dynamic,

… and more higher order specifiers

function (sched, clock, next) { // ... do something next(sched, clock, sched.stop); // Go to the next one (optional). }

DEMO

“Steller Explorer”

Upcoming goodies

currentTime

currentPlayTime

output latency

audio block

Precise coordination with other events

Audio workers

Main JS thread Audio thread

JS

JS

JS N

UI a

nd o

ther

com

pute

sWorkers run JS here!

Native code

To not conclude …

• The Web Audio API is sweet for music and synthesis work … with some care … and it’s going to get sweeter.

• Consider the creative possibilities leveraging WebRTC, WebGL, Web MIDI API (for music and lighting control), Device API, Canvas, SVG, DOM+CSS, asm.js …

Thank you

Questions?

References and Credits

1. ZX Spectrum image from Wikimedia commons 2. BBC Micro image from Wikimedia commons 3. Max patch image from abstrakt.vade.info. 4. "SuperCollider screenshot2". Via Wikipedia - (link) 5. "Pure Data with many patches open (showing netpd project)"

by Reduzent - This is a screenshot of a netpd session. Licensed under CC BY-SA 3.0 via Commons - (link)

6. GW-BASIC screenshot from Wikimedia commons 7. Peter Kellock is the original creator of the Rhythm Engine 8. Steller - https://github.com/srikumarks/steller