[coscup 2013] audio competing

Web Developer or OS Developer?

Experience from FirefoxOS

Alive COSCUP 2013

me @alivedise http://alivedise.github.io http://github.com/alivedise alive@mozilla.com alegnadise@gmail.com ~2.5year Frontend Exp.

聽說COSCUP一定要放貓

Operating System

?

http://www.flickr.com/photos/klaikong/6041270516/

Operating System

FirefoxOS….OS?

Storage Management

fileSystem(yet)

DeviceStorageWatcher

Memory Management

Page Visibility

mozAppType

User Interface

Multi Window Management

Statusbar

System Overlay

App Overlay

Web Activity

Keyboard

Hardware Resource

Management

Wi-Fi

Radio

GPS

Bluetooth

Power

CPU WakeLock

Audio/Media

Multi-Tasking

mozBrowser remote=true

Card view

• Multi Channel • Audio Competing • Background Play

Audio Management

The beginning of everything..

http://bugzil.la/805333

805333

Why do we need multi- channel on OS?

Sounds within FxOS - <audio/>

Sounds within FxOS - radio

Sounds within FxOS - alarm

Sounds within FxOS - voice

The world of single channel..

Peace

But..

Too peaceful.

This means:

When you mute the device, your alarm also is muted. So..

Overslept ♪♪♪~

Zzzzz..

How does OS avoid that?

Different settings for audios on different purposes.

On the other hand..

Competing

On Desktop browser, we don’t need competing.

How about mobile OS?

Cases • one <audio/> in one app. • two <audio/> in one app. • two <audio/> in two apps. • one <audio/> and one not-audio-sound in two apps.

It’s all about UX.

User wants to listen music in background.. User has difficulty to mute the background app.. User needs the alarm to sound anyway..

Android solution: audio focus • http://developer.android.com/training/managing-audio/audio-focus.html

• App is responsible to request the audio focus on demand and release the focus actively.

Problems • How to request audio focus in web? • Volume control of different type of audio source. • Telephony is not using audio tag. • FM is not using audio tag. • …

Proposal • New attribute for media tags. • Background play and competing needs permission check.

• Expose current occupied media info to system.

Page Visibility • HTML5 Page Visibility • The page visibility state is simple on desktop browsers by toggling tabs.

• A little complicated on FirefoxOS • Window(App) has opening/closing transitions. • Screen off means page visibility is hidden, too. • Background page has higher possibility to be killed. (Memory Management!)

mozAudioChannelType

telephony alarm

notification content

normal(default)

Multi channel volume settings

• ringer, desktop-notification

audio.volume.notification

• music, FMRadio

audio.volume.content

• alarm

audio.volume.alarm

• telephony

audio.volume.voice

• Bluetooth SCO

audio.volume.bt_sco

Usage

OR

manifest.webapp

app.html

app.js

Interrupted Event

Current Competing Policies •  Channel with higher priority interrupts the

lower channel. •  Foreground page is always playable. But

interruption occurs or not depends on the channel of the media element.

•  When screen is off, the current app or page is not brought to background now if it’s playing normal channel audio now. (Memory and Power concern!)

•  Audio elements being interrupted could know by ‘mozinterruptbegin’ and ‘mozinterruptend’.

content� normal�

Gecko sidemozAudioChannelType

audio

App

nsIDOMHTMLAudioElement

nsIDOMHTMLMediaElement

nsHTMLMediaElement

nsIAudioChannelAgentCallback

nsIAudioChannelAgent

nsBuiltinDecoder

nsMediaOmxDecoder

nsMediaDecoder

DataSource

MediaStreamSource

MediaResource

- SetReadMode()

create

IOMX

OMX

MediaExtractorOMXCodec

MediaSource - Tell()- Seek()- Read()- GetLength()

hw codec

sw codec

MediaBuffer

- CanPlayChanged()

- "mozinterruptbegin" event- "mozinterruptend" event

OmxDecoder

nsHTMLAudioElement

nsIChannel

create

nsMediaOmxReader

nsBuiltinDecoderReader

create

create

MediaOmxStateMachine

nsBuiltinDecoderStateMachine

nsDecoderStateMachine

create

nsIThreadnsIThread

control

create

nsDocument

- NotifyOwnerDocumentActivityChanged()

FileMediaResource

ChannelMediaResource

nsAudioStream

nsNativeAudioStream

libsydneyaudio

AudioTrack

AudioFlinger

- Write()- Pause()- Resume()- IsPaused()- Drain()- GetMinWriteSize()- GetPosition()- SetVolume()

platform: Firefox OS(gonk)branch: Firefox OS 1.01 (b2g18)

will be replaced to libcubeb in v1.2

Audio Streams Diagram

https://github.com/sotaroikeda/firefox-diagrams

Since API is implemented..

World peace?

No…

Don’t forget the dependency tree.

Incoming Issues • Silent mode under multi channels • Background play ability of existing web pages • Background play and competing for video • Short playing sound is hard to adjust. • DTMF tones • Power consumptions. • Bluetooth earphone has its own profile. • …

Policy for Silent mode • Easy for single channel • Multi channels

• The default channel for volume rockers is ‘notification’. (Same as Android)

• Silent mode interesting channels • notification • content/normal

• Alarm is not affected by silent mode.

Policy for video element • <video> doesn’t need to play at background.

• Currently, use ‘visibilitychange’ event. • <video> by default shall be able to compete the background playing content channel. • Separate background play and audio competing for video element.

Background play issues • Existing web pages: “Hey, I’m a music site!”

• grooveshark • youtube • …

• How to play when screen is off without adding mozAudioChannelType to each web site? • Depart screen off state from invisible page visibility state.

CE Certification • EN 50332-1 • Output level through earphone

should be under 85 dBA. • Prompt a warning when earphone is

plugged and the volume is greater than 85 dBA.

• The cool down time of CE warning popup is continuous 20 hours music listening.

OS level policy for CE • Only apply to content/normal channel. • Output power is hardware-dependent. Exactly, we don’t know the db value for specific devices. • The CE maximum value is customizable for different OEM via default settings.

After all..

UX logic in backend and frontend���

backend

frontend

Find out solutions for existing web sites without new API.���

• As a platform, it’s difficult to make decisions. • New API takes time to be mature.

The scope is undefined before you look into it. ���

To be honest, I wrote few code in these issues. But lots of online/offline discussions.

The position of Frontend

https://speakerdeck.com/josephj/f2e-the-keystone

• It doesn’t matter you are making a web site, a web app or an operating system.

• Frontend is the one connecting UX and Backend, and struggle for ideal design and real requirement.

Thanks! Not the end, to be continued..