A Roadmap:A Roadmap:Beyond Big API’sBeyond Big API’s

Why Software Component Frameworks Why Software Component Frameworks

May Be Interactive Audio’s FutureMay Be Interactive Audio’s Future

Chris Grigg chrisg@control-g.com

ThesisThesisBig audio API’s compromiseBig audio API’s compromise

delivered audio qualitydelivered audio quality

• • To use them, you have to call their functions; To use them, you have to call their functions;

• • Every API uses a different interface;Every API uses a different interface;

• • So Mixing or Changing APIs is too hard.So Mixing or Changing APIs is too hard.

• • Many custom API’s & media types have no tools.Many custom API’s & media types have no tools.

This sucks because:This sucks because:

• • No single API does everything well, or runs on all platforms.No single API does everything well, or runs on all platforms.• Time invested in custom API features doesn’t port.• Time invested in custom API features doesn’t port.

• • Not about specific faults of particular API’sNot about specific faults of particular API’s

• • A Structural Problem of all big audio API’sA Structural Problem of all big audio API’s

• • Programming itselfProgramming itself the bottleneck. the bottleneck.

ThesisThesis

We want to be able to construct runtime We want to be able to construct runtime audio systems out of arbitrary functional audio systems out of arbitrary functional blocks from multiple vendorsblocks from multiple vendors

Our Studios Have Many Different Synthesizer BrandsOur Studios Have Many Different Synthesizer Brands

Because They’re All Good At Different Things –Because They’re All Good At Different Things –

Our Runtime Systems Need Similar DiversityOur Runtime Systems Need Similar Diversity

In this ‘What If’ world:In this ‘What If’ world:

• • Units of software organization become SmallerUnits of software organization become Smaller

• • They become ModularThey become Modular

• • Each Unit Takes On a Specific FunctionEach Unit Takes On a Specific Function

• • Units Have to be able to Talk to one anotherUnits Have to be able to Talk to one another

An Architecture Fundamentally DifferentAn Architecture Fundamentally Differentfrom our current big, do-everything from our current big, do-everything

Audio API development & use practices.Audio API development & use practices.

Things We NeedThings We NeedTo Get ThereTo Get There

11 Software Component Framework Software Component Framework with Object Model with Object Model

22 Audio Signal Interconnect Mechanism Audio Signal Interconnect Mechanism

33 Control Signal Interconnect Mechanism Control Signal Interconnect Mechanism

44 Scriptable Control Scriptable Control

11

A StandardizedA Standardized

Audio Software Component FrameworkAudio Software Component Framework

with Object Modelwith Object Model

If our audio blocks are components,If our audio blocks are components,

we need a framework to hold them.we need a framework to hold them.

Typical Component Typical Component FrameworkFramework

Framework

SoftwareComponent

A

Manager MaintainsManager MaintainsComponent InstancesComponent Instances& Directs Host Calls& Directs Host Calls

Instances of VariousInstances of VariousComponent ClassesComponent Classes

SoftwareComponent

B

SoftwareComponent

C

SoftwareComponent

D

Host Application

Framework Manager

GUI Authoring ToolGUI Authoring ToolUses Same Framework & Components As RuntimeUses Same Framework & Components As Runtime

Select, Configure, & Connect ComponentsSelect, Configure, & Connect ComponentsSave Configuration for Use at RuntimeSave Configuration for Use at Runtime

Having a Uniform,Having a Uniform,Implementation-IndependentImplementation-Independent

Object Model allows:Object Model allows:

• • Single, Universal Authoring ToolSingle, Universal Authoring Tool

• • Cross-Platform ContentCross-Platform Content

• • Cross-Platform Runtime Communication Cross-Platform Runtime Communication

2 & 3:2 & 3:InterconnectionInterconnection

Audio Signals & Control SignalsAudio Signals & Control Signals

Connecting Anything to AnythingConnecting Anything to Anything

Requires Control & Audio Interconnect Requires Control & Audio Interconnect StandardsStandards

Most Component FrameworksMost Component Frameworks

Don’t Have InterconnectionsDon’t Have Interconnections

to Deal withto Deal with

No off-the-shelf solution –No off-the-shelf solution –

We need a development effortWe need a development effort

2

Audio Interconnection MechanismAudio Interconnection Mechanism

Between ComponentsBetween Components

Audio Signal FlowAudio Signal Flow

Between ComponentsBetween Components

……Doesn’t Require Multiple BuffersDoesn’t Require Multiple Buffers

Audio Signal FlowAudio Signal Flow

Using Single BufferUsing Single Buffer

Multiplexed Operations on a Single BufferMultiplexed Operations on a Single Buffer

RepresentingRepresentingInterconnectionsInterconnections

in thein the

Framework Object ModelFramework Object Model

Block Diagram ViewBlock Diagram View

Object Model View:Object Model View:Components and ConnectionsComponents and Connections

Everything derives fromEverything derives fromclass Componentclass Component and and class Connectionclass Connection

Benefits I

• • Better Programmer/Soundmaker CommunicationBetter Programmer/Soundmaker Communication

• • Increases Clarity of PurposeIncreases Clarity of Purpose

• • Simpler, Faster Code DevelopmentSimpler, Faster Code Development

• • Promotes Audio API ProliferationPromotes Audio API Proliferation

• • Encourages View of Audio APIs as InterchangeableEncourages View of Audio APIs as Interchangeable

• • Simplifies Porting of Game Engines and TitlesSimplifies Porting of Game Engines and Titles

• • Encourages Porting Components to New PlatformsEncourages Porting Components to New Platforms

A Message-Based MechanismA Message-Based Mechanismfor Control Signal Interconnectionfor Control Signal Interconnection

Between Software ComponentsBetween Software Components

33

Uh… why “control signals”?Uh… why “control signals”?

To avoid messes like this.To avoid messes like this.

Rat’s Nest of Function CallsRat’s Nest of Function Calls

Like MIDI CablesLike MIDI Cables

for Software Componentsfor Software Components

Unified Control Signal – Single-Point ConnectionUnified Control Signal – Single-Point Connection

Assemble Controlled Sound SystemsAssemble Controlled Sound Systems

from Software Componentsfrom Software Components

Like a Recording Studio:Like a Recording Studio:Audio Connections & Control ConnectionsAudio Connections & Control Connections

Host Game UsesHost Game UsesSame Control MechanismSame Control Mechanism

Still Like a Recording Studio:Still Like a Recording Studio:Audio Connections & Control ConnectionsAudio Connections & Control Connections

Components Can ProcessComponents Can ProcessControl SignalsControl Signals

(not just audio)(not just audio)

Examples: Y-Cord, Merger, AdaptorExamples: Y-Cord, Merger, Adaptor

Easy Communication withEasy Communication withExternal Devices Using Same ModelExternal Devices Using Same Model

Consistent Binary Control Signal FormatConsistent Binary Control Signal FormatMakes This PossibleMakes This Possible

Multi-Component ‘Templates’Multi-Component ‘Templates’

Add Configuration PowerAdd Configuration Power

Example: Mixer Channel Strip TemplateExample: Mixer Channel Strip Template

Many Components Instantiated in Single StepMany Components Instantiated in Single Step

with Internal Routing, Configuration, Initial Settingswith Internal Routing, Configuration, Initial Settings

Control Signals Give Us KnobsControl Signals Give Us KnobsBoth the Programmer Both the Programmer andand the Soundmaker the Soundmaker

So what kinds of messagesSo what kinds of messagesare in the Control Signals?are in the Control Signals?

Standardize theStandardize theControl Signal ConnectionControl Signal Connection

But But NotNot the Command Set, the Command Set,Parameter Set, etc.Parameter Set, etc.

Arbitrary MessagesArbitrary MessagesOn The ConnectionOn The Connection

How Do You Know What MessagesHow Do You Know What Messagesa Component Accepts?a Component Accepts?

You Ask It.You Ask It.

Don’t Predefine the Message Vocabulary.Don’t Predefine the Message Vocabulary.

Let Every Component Define Its Own.Let Every Component Define Its Own.

Every Component HasEvery Component Has

Its Own Command SetIts Own Command Set

…And …And PublishesPublishes It It

This Is How The GUI Authoring ToolThis Is How The GUI Authoring ToolKnows What Each Component Can Do.Knows What Each Component Can Do.

Simple Piano ComponentSimple Piano ComponentTwo Control InputsTwo Control Inputs

Simple ExampleSimple Example

Simple Piano ComponentSimple Piano ComponentControl Interface DescriptionControl Interface Description

ControlInput1:Name: MIDI InProtocol: MIDI Messages

Interpretation: (MIDI Implementation Chart appears here) ControlInput2:

Name: Sustain Switch InProtocol: Boolean MessagesInterpretation: low bit 1 = pedal down, 0 = up

Ask the Component for its Interface Description and you get a Ask the Component for its Interface Description and you get a list of the Messages that its Control Inputs Understandlist of the Messages that its Control Inputs Understand

3D Panner Component3D Panner ComponentOne Control InputOne Control Input

More Complex ExampleMore Complex Example

3D Panner Component3D Panner ComponentControl Interface DescriptionControl Interface Description

Commands:SetVolume( Fixed16 ) -- Master out volumeMute( Boolean ) -- Mute (TRUE) or Pass audio

SetSpatialPosition( Std3DVector ) -- 3-space

The Messages that the Control Input UnderstandsThe Messages that the Control Input Understands

But Weren’t We Trying to Get AwayBut Weren’t We Trying to Get Away

from Function Calls?from Function Calls?Commands:

SetVolume( Fixed16 ) -- Master out volumeMute( Boolean ) -- Mute (TRUE) or Pass audio

SetSpatialPosition( Std3DVector ) -- 3-space

This looks just like a Big Audio API!This looks just like a Big Audio API!

True, this semantic is similar to the HLL function calls...True, this semantic is similar to the HLL function calls...

But it’s expressed differently at runtime.But it’s expressed differently at runtime.

SetVolume( Fixed16 ) Mute( Boolean )

SetSpatialPosition( Std3DVector )

Not as compile-based function calls, but Not as compile-based function calls, but rather inrather in Data-Based Messages Data-Based Messages that can be that can be

routed dynamically at runtime.routed dynamically at runtime.

How do Components Publish theirHow do Components Publish theirInterface DescriptionsInterface Descriptions

at Runtime and Authoring Time?at Runtime and Authoring Time?

Some Invention Is Required Here.Some Invention Is Required Here.

Learn from existing solutions to similar problems:Learn from existing solutions to similar problems:• • COMCOM

• • CORBA’s IDLCORBA’s IDL

• • Apple Event Terminology ResourcesApple Event Terminology Resources

• • XML’s DTDXML’s DTD

• • MAXMAX Objects Objects

44

Adding theAdding the

Scripting LayerScripting Layer

Scripted Control of allScripted Control of allAudio Components & MediaAudio Components & Media

With Enough Component Types and Media Types, With Enough Component Types and Media Types,

the ‘Ultimate Interactive Audio Architecture’the ‘Ultimate Interactive Audio Architecture’

Sound Dept. DeliversSound Dept. DeliversPlatform-IndependentPlatform-Independent

Media & ScriptsMedia & Scripts

Components, FrameworkComponents, Framework& Script Player Get& Script Player GetPorted to All PlatformsPorted to All Platforms

Game UsesGame UsesSame Sound Event CuesSame Sound Event CuesOn All PlatformsOn All Platforms

Benefits IIBenefits II

• • Simplified Control of Sound From GameSimplified Control of Sound From Game

• • Increased API Experimentation FlexibilityIncreased API Experimentation Flexibility

• • Flexible Patching of Control Sources & DestinationsFlexible Patching of Control Sources & Destinations

• • Encourages Plug-In Style DevelopmentEncourages Plug-In Style Development

• • Ability to Transform Game Control VariablesAbility to Transform Game Control Variables

• • Sound Designer Control over InteractivitySound Designer Control over Interactivity

• • Auto-Porting of Auto-Porting of InteractiveInteractive Audio Content Audio Content

ConclusionConclusionAn essentially non-commercial idea:An essentially non-commercial idea:

• • The limits of big audio APIs are structural.The limits of big audio APIs are structural.

• • They favor programmer contributions while They favor programmer contributions while marginalizing soundmaker contributions – through marginalizing soundmaker contributions – through poor or missing tools, but also by virtue of their poor or missing tools, but also by virtue of their fundamental compile-based architectures. fundamental compile-based architectures.

• • This dynamic hurts the delivered audio quality.This dynamic hurts the delivered audio quality.

• • Open standards for component software appear to Open standards for component software appear to offer a way out, and might serve as the basis of a offer a way out, and might serve as the basis of a generalized interactive audio methodology as generalized interactive audio methodology as universal and stable as MIDI has been. universal and stable as MIDI has been.

Because Media is a Harsh Master.Because Media is a Harsh Master.