hh qualcomm ‘hear’ & now qualcomm® snapdragon™ audio

‘Hear’ & Now Qualcomm® Snapdragon™ Audio

Ravi Satyanarayanan, Director Product Management, Qualcomm Technologies, Inc. Tony Rodriguez, CTO, Digimarc Richard Kjerstadius, Director, DSP Software, Opalum

Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.

‘Hear’ & Now: Snapdragon Audio

User Experience

1 Ecosystem Enablement

2 Digimarc

Watermark Detection

3 Opalum

Speaker Protection

Audio user experience Enhancing and expanding the phone experience

Camcorder

Entertainment

Natural user interaction

Voice call

Music playback

Voice call. Hear. And be heard. Voice enhancement technology for clearer conversations

• Fluence & Fluence Pro noise cancellation technology echo cancellation and noise suppression

• Freedom to hold the phone how you want

− Landscape or portrait speakerphone

− Handset

• Hear your conversation & not the background sound

− Active Noise Cancellation to suppress noise around you

− Automatically amplify speech relative to background sound

• HD Voice in all calls

• Reduce noise on the incoming calls

Music playback Hear music as the musicians intended

• Hi-Fi audio at the same quality as studio masters

- High quality WCD codec

- End-to-end 24bit / 192kHz

• Listen all day with low power playback on the DSP

• Active Noise Cancellation to hear only what you want to hear

Home theatre on-the-go Enjoy the same surround sound available in home theaters wherever you are

• Virtual surround sound over headphones to bring the immersive experience with you

• Big booming sound

- Bigger bass, clearer vocals, louder sounds

• Collaborations with industry leaders: Dolby, DTS

Camcorder Capture what you want to hear

• Capture what you see with Sound Focus

- Suppress unwanted noise from outside of the field of view

• Audio zoom when the camera zooms in

• Surround Sound Record for the Full HD / 4K camcorder experience

• High quality audio, with high dynamic range

Voice natural user interface

• Always on, low power voice activation

- No need to press a button to interact with your device

- Launch applications and give commands quickly

• Activate & Unlock in one step

- No need to enter a PIN

• Hands free operation with Fluence

- Drive mode

User Experience

1 Ecosystem

Enablement

2 Digimarc

Watermark Detection

3 Opalum

Speaker Protection

Audio ecosystem

• Added on Snapdragon 810, 808

• Great for low latency, timing sensitive algorithms

• Speaker Protection by Opalum

Audio front-end Voice

3rd-party relationships for OEM differentiation

• End-to-end high-fidelity audio

– Native 24bit/192kHz framework

• Low power decode on Qualcomm® Hexagon™ DSP

• Snapdragon Voice Activation

• ISVs can enable similar solutions using the framework

Always on, low power

• DSP-accelerated voice codecs and ECNS for VoIP

• Hexagon Access Program support to add voice codecs and processing modules

Qualcomm Hexagon is a product of Qualcomm Technologies, Inc.

How others add value, and find success

User Experience

2 Digimarc

Watermark Detection

3 Opalum

Speaker Protection

Qualcomm® Snapdragon SenseTM Audio + Digimarc Discover

Enabling Low-Power Always-On Contextual Awareness

Tony Rodriguez

Qualcomm Snapdragon Sense is a product of Qualcomm Technologies, Inc.

Digimarc

Vision and Mission • Enable mobile devices to see, hear, and understand by providing

digital identities for all media objects (Internet of Everything) Approach

• Lessons learned from the last decade of deployment (started w/DoJa & BrewTM):

• ‘Seeing & Hearing’ is a function of how fast, accurate and reliable the underlying object/media recognition algorithms are.

• Deterministic object recognition is significantly quicker, more accurate and reliable.

• Developers are best served when fast, reliable and accurate object recognition is available as a system service that minimizes bandwidth consumption and is always-on & low-power.

• 2013: Delivered Intuitive Computing Platform (ICP) to our retail, publishing and brand partners who have deployed apps to enable engagement with print, packaging, radio, etc.

BREW is a product of Qualcomm Technologies, Inc.

Digimarc Discover built on ICP: Enabling Shoppers Journey

• Digimarc® Discover app & SDK’s built on Intuitive Computing Platform.

• Enables object and media

recognition through all phases of Shopper’s Journey

• Discovery & transaction via mobile devices across all touch points of the Journey

• Digimarc® Barcode: Imperceptible

barcodes for CPG that can double throughput at front-of-store.

Examples: Publications, TV, Retail

Print to Digital Contextual Second-Screen Shopping

Deterministic Identification Powered by Digimarc ID’s

We can’t see or hear Digimarc ID’s, but mobile and wearable devices can

Digimarc ID

Original Digital Media

Digimarc ID Imperceptibly Embedded

Enabled Content

Music, Movies or Images

Television and Radio

Comparison of App Experience in Noisy Environment

• Digimarc Audio ID’s

effective in hostile environments.

• Accuracy maintained, no missed matches (deterministic)

• Developers can assign multiple ID’s, creating richer, contextually aware experience.

Digimarc Audio ID: Details

• Signal Construction • Weak signal in 120 Hz to 8 kHz

range, communicated via host audio (channel)

• Extensible & key based payload protocols

• Embedding

• Psychoacoustic principles and empirical analysis used to hide watermark

• Ambient Detection

• Spectral Processing • Payload Recovery

Technique Frequency Domain WM

Description Frequency Domain Spread Spectrum

Characteristics Watermark signal added to audio spectrum (frequency magnitude)

Payload Extensible 24 bit

Audio required for detection

1-3 Seconds

Porting to the Qualcomm® HexagonTM

Goal: People don’t have “ear-lids” – sound is the most pervasive and rich source of context in the human experience, our sub-conscious is continually (even when asleep) categorizing audio and applying policies on when and how to engage our consciousness. Mobile & wearable devices should do the same!

• Phase 1: Re-factoring (pipeline architecture), Fixed Point, Memory Reduction • Phase 2: Hexagon Port of Detector • Phase 3: Low-Power Activator Implementation

• Result: Developers able to create always-on contextually aware applications that are fast, reliable and accurate.

Qualcomm Hexagon is a product of Qualcomm Technologies, Inc.

Phase 1: Re-factoring Detector (Pipeline Architecture)

• Mobile implementation optimized for OS & application processor

• Each block re-coded, reducing memory consumption and functional granularity to aide in porting

• 8MB -> 256K Runtime

FFT Intermediate Accumulation Magnitudes

Spectral Filter

Final Accumulation Magnitudes

Normalize Magnitudes

For every 512 samples @ 16 Khz For each 1.024 second

CRC SNR Gating Decode Select Decode Candidates

Version Detection

For each 1.024 second For each 1.024 second + Scaling

Digimarc ID

Spectral Processing

Payload Recovery

Phase 2: Hexagon Port

Detector Application

Hexagon & Audio Codec

• Application reliant on High Level Operating System (HLOS) to retrieve audio samples and deliver to Detector

• Audio sampled at 16Khz but processed only once a second.

• Difficult for Application Processor cores to enter low-power state.

Application Processor

Digimarc Qualcomm Technologies Hexagon SDK

Phase 2: Hexagon Port

Detector Stub Application

Fast RPC

Detector

• Hexagon SDK to Port detector C++ source.

• Significantly reduced load

on Application Processor cores

• Utilized library of optimized functions for Hexagon (FFT, etc.)

• QAIC handled complexity of FastRPC marshaling

Phase 3: Low-Power Activator

Fast RPC

“Activator” S

ervice & P

olicy Engine

• Worked with Qualcomm Technologies to

build Activator and route audio from Codec • Eliminated need for HLOS to manage audio.

Detector

Fast RPC

Detector Codec

“Activator” S

ervice & P

olicy Engine

• Policy Engine able to wake Application

Processor and invoke appropriate application when Digimarc Audio ID detected

• Network bandwidth consumed after object identification.

Policy Engine

Fast RPC

Detector Codec

“Activator” S

ervice & P

olicy Engine

• Policy Engine able to wake Application

Processor and invoke appropriate application when Digimarc Audio ID detected

• Network bandwidth consumed after object identification.

• Result: Application Processor & RF allowed

to enter low-power state.

• Developers able to create always-on contextually aware applications that leverage the ubiquity of audio in our everyday lives

Policy Engine

Conclusion : Snapdragon Sense Audio + Digimarc Audio ID’s

• People don’t have “ear-lids” – sound is the most pervasive and rich source of context in the human experience, our sub-conscious is continually (even when asleep) categorizing audio and applying policies on when and how to engage our consciousness.

• Mobile and wearable devices are now able to do the same, remain always aware and wake the

Application Processor as needed. • Able to maintain an always-on posture with no network dependencies – ramps power and

bandwidth consumption after identification has occurred (unique to Digimarc ID’s). • Enabled by Snapdragon Sense Audio & Digimarc Audio ID’s that provide accurate, reliable context. • Result: Accurate and application specific context can be incorporated in all sources of audio.

• Applications enabled for developers: Retail, Second-Screen, Indoor Location, Radio, Concerts, etc.

‘Hear’ & Now: Snapdragon Sense Audio

User Experience

2 Digimarc

Watermark Detection

3 Opalum

Speaker Protection

High-End audio from constrained devices

Agenda

• Trends in Mobile Audio • Opalum Overview and Tech • Collaboration with Qualcomm Technologies • What’s next?

Trends in Mobile Audio

• Maximizing loudness and bass – Power doesn’t kill a speaker – Pushing the limits on thinner form-factors

• Multi-channel audio goes mobile – Virtualization techniques for movies and games. – Parametric audio

• Reducing power consumption – “Always on” features demand DSP core offloading – Increasing level of integration into main chipset – Other ways of reducing power consumption?

Overview and Tech

Collaborating with Qualcomm Technologies

• Seamless integration between LabSuite and Snapdragon • Audio tuning lead-time drastically reduced • Working with the Hexagon tools • Dynamic loading enables really cool android app possibilities

What’s Next?

For more information on Qualcomm, visit us at: www.qualcomm.com & www.qualcomm.com/blog

©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Snapdragon, Hexagon, Brew and Fluence are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Uplinq and Snapdragon Sense iaretrademarks of Qualcomm Incorporated. All Qualcomm Incorporated trademarks are used with permission. HY-FI is a trademark of Qualcomm Atheros, Inc., registered in the United States and other countries, used with permission. Other products and brand names may be trademarks or registered trademarks of their respective owners.of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.

Thank you FOLLOW US ON:

hh qualcomm ‘hear’ & now qualcomm® snapdragon™ audio

affiliated companies

opalum qualcomm snapdragon

background sound hd

unwanted noise

sound focus

dsp active noise cancellation

virtual surround sound

surround sound record

Devices & Hardware

qualcomm snapdragon benchmark report

pm80-lr irda version powerful qualcomm snapdragon will free...

qualcomm snapdragon navigator™ - qualcomm …® snapdragon...

qualcomm embedded computing use cases tm 810 · qualcomm...

qualcomm snapdragon 600-based smartphone

alenwu€¦ · with qualcomm® snapdragon™ 845 mobile...

zte t22 - s3.amazonaws.com and google calendar ......

snapdragon 888 camera presentation - qualcomm

qualcomm snapdragon 835 soc 介紹

qualcomm® snapdragon sense™ id 3d fingerprint …. a...

qualcomm snapdragon mobile platform opencl general...

qualcomm snapdragon 800, tsmc 28hpm process

snapdragon profiler user guide - qualcomm developer network

qualcomm snapdragon automotive solutions

dragonboard™ based on qualcomm® snapdragon™ …

snapdragon™ heterogeneous compute sdk...qualcomm®...

qualcomm® snapdragon™ profiler quick start guide

qualcomm snapdragon s4 pro-based smart phone(simple)

qualcomm user experiences snapdragontm 660...qualcomm...

ubuntu linux on the qualcomm snapdragon 600 processor