java video and audio in consumer devices: jmf and mm api

T-111.500 Multimedia SeminarMobile Internet Technology Architecture (MITA)

Tietoliikenne- ja multimediatekniikan laboratorio (TML)

Java Video and Audio in Consumer Devices: JMF and MM API

Pablo César and J. Luc Lamadon

{pcesar, lamadon}@tml.hut.fi

Helsinki University of Technology



Outline

• Introduction

• Java Overview

• Personal Computer

• Digital Television Receiver

• Mobile Phone

• Conclusions

• References



Introduction (1/3)

• What is multimedia?– Multiple media (e.g., audio, video, images, graphics,

text) or multimedia objects – Synchronisation of multimedia elements– Interaction between user and system

• Multimedia in Consumer Devices, user expectation– Latest mobile phones with camera, MMS, Internet

connection, streaming– Digital TV based on audio visual stream + interactive

services– PC with 3D games, videoconference, streaming



Introduction (2/3)

• Video objects in programming tools:– Synchronized Markup

Integration Language (SMIL)

– Flash presentations

– Java Media Framework (JMF)

• Stand alone video player:– Proprietary players:

Real One, Windows Media Player...

– Open Source players: winamp, MPlayer...

Audio and Video in Consumer Devices, two alternatives:



Introduction (3/3)

• The focus of this presentation is video objects in a programming tool

• Specifically Java, why?– Consumer devices includes Java

• Digital TV = Multimedia Home Platform (MHP) and JDK 1.1.8

• PCs = Java 2 Standard Edition• Mobile phones = Connected Limited Device

Configuration (CLDC) + Information Device Profile (MIDP)

– Independent of Operating System



Java Overview (1/2)



Java Overview (2/2)

Decodes and plays the media data

Retrieves the actual media data

Implements the state machine



Personal Computer (1/4)

• Services:– Internet, videoconference– Games– …

• Physical Characteristics:– Pointer device (e.g., mouse) and keyboard as major

input mechanisms– Screen resolution: 640x480 to 1600x1200 pixels– Runtime memory: around 128 to 256 MB

• Java Environment– Java 2 Standard Edition (J2SE) including Abstract

Windowing Toolkit (AWT) and Swing




• JMF as an optional package either version 1.0 or 2.0

• JMF relies on the mentioned classes: Manager, Player, Data Source, and Controller.

• Player behaviour:– Unrealised: when it does not have all the information to

acquire the needed resources– Realised: when it has all the information to acquire the

needed resources– Prefetched: when it has all the needed resources, and has

already prefetched enough media data to start playing immediately

– Started: when it is actually playing the media




• Time model: Clock defines the basic timing and synchronisation operations, which control the presentation of media data (e.g. forward, reward)

• Data model: DataSource encapsulates both the location and protocol of media

• JMF 2.0 includes as well a low level API– A Processor performs user-defined processing on the

media data using JMF plug-ins (e.g. Codec, Demultiplexer, Effects, Multiplexer, Renderer)

• Render is done in an AWT Component:– Player.getVisualComponent()– Player.getControlPanelComponent()



Digital Television Receiver (1/5)

• Multimedia platform in the living room• Services:

– Audio visual stream (normal passive watching)

– Interactive services (active behaviour)

• Physical Characteristics– Remote control as major input mechanism

– Screen resolution: 720x576 pixels (minimal)

– Runtime memory: at least 16 MB

• Java Environment– MHP




• MHP includes JMF 1.0 (obviously, not as an optional package) with some clarifications, restrictions, and extensions

• The basic classes are again: – Manager, Player, DataSource, and Controller

• The player behaviour is the same as in JMF 1.0:– Unrealised, Realised, Prefetched, and Started

• JMF can be used both for broadcast stream and local files (only in high-end receivers)

• Video drips presents I-frame and consecutive P-frames (high quality interactive graphics)




• Locators (reference to a particular piece of media) basically three main types:– javax.media.MediaLocator (and subclasses): media clip– org.davic.net.Locator (and subclasses): DVB transport

stream or service– javax.tv.locator.Locator: any type of digital TV system

• Time based (Clock) is useless in broadcast– Solution to synchronise = stream events

• A JMF player cannot access directly services in other transport streams (unless explicitly done)

• Includes new player events (e.g.,):– Media Presented: The media actually starts showing




• Includes a number of specific digital TV controls:– Media Select: changes the actual stream presented (e.g.,

change the angle of a camera)– Language: intended to control the audio and subtitles

(if present) language

• Television display is composed of:– Background + Video + Graphics Layers

• Video media is, normally, hardware decoded and demultiplexed. So, rendered in video layer (no AWT container)– getVisualContent() returns null– getControlPanelComponent() returns null




• Still, video can be controlled:– BackgroundVideoPresentationControl: location

and size– VideoFormatControl: picture format and aspect

ratio (e.g., 4:3)

• Maybe in the future (high-end receivers) a real integration of media in graphics layer can be obtained.

• External audio clips can be played in JMF-MHP



Mobile Phone (1/5)

• Mobile phones are becoming powerful Multimedia platforms

• Provide a number of services:– Internet (e.g., Nokia + Opera), MMS, Video Player– Office capabilities

• Physical characteristics– Key Pad as major input mechanism– Screen resolution: 84x48 to 120x130 pixels– Runtime memory: 160 to 512 KB

• Java Environment– Mobile Information Device Profile (MIDP) version 1.0

or 2.0



Mobile Phone (2/5)

• Mobile Multimedia API (MMAPI) version 1.0• Latest release was on 26th June, 2003 http://

java.sun.com/products/mmapi/ • MMAPI Description:

– It extends MIDP functionality by providing audio, video and other time-based multimedia support

– It is a thin Java layer completely platform dependent– It is not JMF– It is an optional package

• MIDP 2.0 includes the audio-only subset from MMAPI (i.e. Audio Building Block)



Mobile Phone (3/5)

• Same concepts as JMF: Player, Controller, Manager and DataSource.

• Player behaviour same (Unrealised, Realised, Prefetched, Started) as JMF but one more state:– Closed: the player cannot be used again, it has

released most of the resources

• Controller is embedded in Player class



Mobile Phone (4/5)



Mobile Phone (5/5)

• The types of media supported depends on the controls associated to the Player– Player.getControls() returns the supported controls– Some examples: VolumeControl, ToneControl…

• The Player is content and protocol agnostic• A Player renders media data in a component

dependent on the device configuration, two options:– AWT Component– MIDP Canvas or Items

• VideoControl manages the location and the size of the video



Conclusions (1/2)

• The actual capabilities of the targeted device is the cause of the differences between standards

• Low level versus high level control of the media:– In MM API and JMF (MHP) the actual control of the

media is done at the native level since they are resource-constrained devices (e.g. decoding)

– JMF uses two profiles, JMF plug-ins enables developers to process the data (e.g. multiplexing)

• Behaviour of the player– All the standards have the same player behaviour. But

MM API defines a Closed state to make explicit that all the resources are freed



Conclusions (2/2)

• MM API is influenced by the design of JMF, and have number of similarities: Manager, Player, Data Source, and Controller concepts.– MM API hides Controller within Player

• Video render– JMF = AWT Component (complete integration)– MHP = A layer (transparency can be applied) – MM API = MIDP canvas (minimal integration)

• Controls are different depending on the targeted device (e.g., subtitles language in television)

• MHP includes specific television requirements (e.g., Clock not needed in broadcast, Locators)



References (1/2)

• Multimedia – P. Vuorimaa, Multimedia Technology Course’s Slides, Helsinki

University of Technology, accessed on November 7th 2003, http://www.tml.hut.fi/Opinnot/T-111.350/index_uk.html

– S. V. Raghavan, and S. K. Tripathi, Networked Multimedia Systems: Concepts, Architecture, and design, Upper Saddle River (NJ), Prentice Hall, 1998

• Digital Television– ETSI TS 102 812 v1.1.1, Digital Television Broadcasting (DVB):

Multimedia Home Platform (MHP) Specification 1.1, ETSI, November 2001

– Steven Morris, The Interactive TV Web, http://www.mhp-interactive.org , 2002



References (2/2)

• PC– R. Gordon, and S. Talley, Essential JMF: Java Media Framework,

Upper Saddle River (NJ), Prentice Hall, 1999– Sun Microsystems, JMF Home Page, accessed on November 7th

2003, http://java.sun.com/products/java-media/jmf/index.html

• Mobile Phones– Sun Microsystems, J2ME: Java 2 Platform Micro Edition,

accessed November 7th 2003, http://java.sun.com/products/j2me/ – Sun Microsystems, CLDC, accessed November 7th 2003, http://

java.sun.com/products/cldc/ – Sun Microsystems, Mobile Information Device Profile (MIDP),

accessed November 7th 2003 http://wireless.java.sun.com/midp/ – Sun Microsystems, Mobile Media API (MMAPI), accessed

November 7th 2003, http://java.sun.com/products/mmapi/



Thank you!

Questions?

Comments?

java video and audio in consumer devices: jmf and mm api

Documents