7133238-jmf
Post on 09-Apr-2018
225 Views
Preview:
TRANSCRIPT
-
8/8/2019 7133238-Jmf
1/59
-
8/8/2019 7133238-Jmf
2/59
JMF Media Processing Model
-
8/8/2019 7133238-Jmf
3/59
Media Streams
Often contain multiple channels Tracks
Example MPEG-1 file usually 2 tracks
Audio track
Video track
Demultiplexing
Multiplexing
-
8/8/2019 7133238-Jmf
4/59
Example
Process an mpeg-1 a/v media stream Transcode video track to H.263
Transcode audio track to GSM
Steps Demultiplex to obtain tracks
Decompress video track
Compress using H.263
Decompress audio track
Compress using GSM Multiplex two tracks
Save to file
-
8/8/2019 7133238-Jmf
5/59
JMF Design Goals
Enable input, processing and output of time-based
media
Provides common cross platform API for
accessing underlying media frameworks
Extensible
support additional content types and formats,optimize handling of supported format, create new
presentation mechanisms
-
8/8/2019 7133238-Jmf
6/59
Supported Content Types
Supported types Not always both decode and encode
Differences between platform independent and dependent
versions
Audio
WAV, GSM, MIDI, etc
Image
JPEG, etc
Video
H.261,H.263, MPEG-1, Quicktime, AVI, etc
-
8/8/2019 7133238-Jmf
7/59
Recording, processing, and
presenting time-based media
-
8/8/2019 7133238-Jmf
8/59
High-level Architecture
-
8/8/2019 7133238-Jmf
9/59
Some JMF Base Interfaces
Clock
Controller
Control
-
8/8/2019 7133238-Jmf
10/59
Time Model
The Clock interface Defines basic timing and synchronization operations
Contains a Timebase
Based on the system clock
time-base time
Simply provides current time
Clock marks time for a particular media stream
media time Current position within a media stream
-
8/8/2019 7133238-Jmf
11/59
Time Model
-
8/8/2019 7133238-Jmf
12/59
Clock
Playback rate How fast the Clock is running in relation to itsTimeBase
Examples:
rate of 1.0 represents normal running time
rate of2.0 means presentation will run at twice thenormal rate
ClockT
ransform Media-time =media start-time+ Rate(time-base time
time-base start-time)
-
8/8/2019 7133238-Jmf
13/59
Example
Example: Have a 20 sec MPEG video stream MediaStartTime= 10 secs,
TimeBaseTime= 3 secs
TimeBaseStartTime= 0 secs, TimeBaseTimeTimeBaseStartTime= 3 secs
Media-time= media start-time+ Rate(time-base time time-base start-time)
So if Rate
=1.0
, MediaTime=
?? Alternatively, if rate = -2.0, MediaTime= ??
-
8/8/2019 7133238-Jmf
14/59
Achieving Synchronization
Example Want to force a video renderer to sync to the
timebase of an audio renderer X = audio_renderer.getTimeBase()
Video_renderer.setTimeBase(X)
Both objects would now use the same source of
time.
-
8/8/2019 7133238-Jmf
15/59
ControllerInterface
Controller Interface Defines basic state and control mechanism for an
object that controls, presents or captures time-based media
Two types of Controller: Players and Processors
(considered later)
-
8/8/2019 7133238-Jmf
16/59
Controllers
-
8/8/2019 7133238-Jmf
17/59
Controller lifecycle
-
8/8/2019 7133238-Jmf
18/59
Controller Events JMF objects can post a MediaEvent
Events posted by a Controller: TransitionEvents
Posted when a controller changes state
Change notification events
e.g. RateChangeEvent
ControllerClosedEvents
Posted when Controller shuts down
Corresponding listener interface for each type of JMFobject that can post MediaEvents
-
8/8/2019 7133238-Jmf
19/59
JMF Event Model
-
8/8/2019 7133238-Jmf
20/59
Controls
Mechanism for setting and querying attributes ofan object
Certain objects expose Controls e.g. often used by PlugIns to provide access to their
Control objects
Examples
FrameRateControl
GainControl Can associate listener for when volume changes
-
8/8/2019 7133238-Jmf
21/59
Key objects in JMF
Managers
DataSources
Players
Processors
DataSinks
-
8/8/2019 7133238-Jmf
22/59
General Managers
Intermediary objects Enables new implementations of key interfaces
4 main types Manager
PlugInManager
PackageManager
CaptureDeviceManager
-
8/8/2019 7133238-Jmf
23/59
The Manager object
Manager object used for instantiating: DataSources,
used to deliver time-based multimedia data
Players,
used to control and render multimedia data
Processors,
used to process data and output the processed data
D
ataSinks, takes a DataSource as input and renders the output to a specified
destination
-
8/8/2019 7133238-Jmf
24/59
The Manager Object
-
8/8/2019 7133238-Jmf
25/59
Data Model in JMF
Data Sources Media players use DataSources to manage the
transfer of media-content
DataSource encapsulates location of media and the
protocol used to deliver the media
Typically Identified by a:
URL, MediaLocator
-
8/8/2019 7133238-Jmf
26/59
Capture
Capture devices represented as DataSources e.g. microphone, video capture board, etc
Devices can deliver multiple data streams e.g. audio and video from a camera
e.g. multiple audio tracks from a recording session
You may then wish a single DataSource to contain
multiple SourceStreams
Manager.createMergingDataSource(SourceStreams)
-
8/8/2019 7133238-Jmf
27/59
Push and Pull Data Sources
Data sources can be categorized according to how
data transfer is initiated
Pull Data-Source Client initiates the data transfer e.g. HTTP and FILE
Push Data-Source Server initiates the data transfer e.g. broadcast and multicast media
-
8/8/2019 7133238-Jmf
28/59
Players
Processes an input stream and renders it at a
precise time
Does not provide any control over the processing
that it performs or how it renders the media data
-
8/8/2019 7133238-Jmf
29/59
Players
Player extends the Controller interface. Has a lifecycle
Sends media events
Player as a MediaHandler player = Manager.createPlayer(myDataSource);player = Manager.createPlayer(myMediaLocator);
player = Manager.createPlayer(myUrl);
-
8/8/2019 7133238-Jmf
30/59
Players
-
8/8/2019 7133238-Jmf
31/59
UI Components
Players provides access to UI Components Player (or Processor) can provide two UI
components
Visual component
Control-panel component
Can retrieve these components using methods:
getVisualComponent()
getControlPanelComponent()
-
8/8/2019 7133238-Jmf
32/59
Player States Continued
Players post transitional events as they move
between states
ControllerListener Is the Player in an appropriate state?
Only certain methods make sense in certain states
e.g. calling getTimeBase method on an unrealized
player gives an error
-
8/8/2019 7133238-Jmf
33/59
Processors
Can also be used to present media data
Specialized type of Player that provides control
over processing performed on the input media
stream
-
8/8/2019 7133238-Jmf
34/59
Processing
-
8/8/2019 7133238-Jmf
35/59
Processor Stages
-
8/8/2019 7133238-Jmf
36/59
Additional Processor States
Two additional stand-by states: Configuring
Configured can use TrackControls
-
8/8/2019 7133238-Jmf
37/59
Processing Controls
For a given track, can control processing
operations performed by the Processor by using
the TrackControlfor that track.TrackControl[] = processor.getTrackControls()
Can explicitly select: Effect, Codec and Renderer plug-ins to use
TrackControl[1].setCodecChain( array_of_codecs )
-
8/8/2019 7133238-Jmf
38/59
Configuring the Processor
Consider using a processor to transcode an
(audio+video) QuickTime movie changing
mpeg video track to h.263p = Manager.createProcessor(dataSource)
p.configure()
p.setContentDescriptor.QUICKTIME
tcs[]
=p.getTrackContro
ls()
Returns an array, e.g.2 track controls
-
8/8/2019 7133238-Jmf
39/59
-
8/8/2019 7133238-Jmf
40/59
Processor Summary
A Processor does not have to output data as a
DataSource, such a processor (i.e. one that
renders the data) is effectively a configurableplayer.
-
8/8/2019 7133238-Jmf
41/59
Data Storage and
Transmission
DataSink Used to read data from a DataSource and render the
media to an output destination
Typical actions
Write data to a file, across a network etc
-
8/8/2019 7133238-Jmf
42/59
Using theD
ataSinkMediaLocator dest = new
MediaLocator(file://newfile.wav);
dsink = Manager.createDataSink(ds, dest);
dsink.addDataSinkListener(this);dsink.open();
p.start();
dsink.start();
Wait forEndOfStream event
Close DataSink and remove Listener dsink.close()
-
8/8/2019 7133238-Jmf
43/59
Example
Applet Movie Player
-
8/8/2019 7133238-Jmf
44/59
Applet Movie Player
Simple Java Applet that demonstrates how to
create a simple media player with a media event
listener. It will play the media clip right away andcontinuously loop.
-
8/8/2019 7133238-Jmf
45/59
Basic Steps
Initialisation Retrieve applets FILE parameter
Use this to locate media file and build
URL
Create Player using the Manager object
Register applet as a ControllerListener
-
8/8/2019 7133238-Jmf
46/59
Steps 1 & 2: Resolving a URL
for the media stream// The applet tag should contain the path to the
// source media file, relative to the htmlpage.
if ((mediaFile = getParameter("FILE")) == null)
Fatal("Invalid media file parameter");try {
// Create a url from the file name and the url
// to the document containing this applet.
if ((url = new URL(codeBase, mediaFile)) == null)
Fatal("Can't build URL for " + media
File);
-
8/8/2019 7133238-Jmf
47/59
Step 3: Using Manager to
Create a Player// Create an instance of a player for this media
try {
player = Manager.createPlayer(url);
}catch (NoPlayerException e) {
System.out.println(e);
Fatal("Could not create player for " + url);
}
-
8/8/2019 7133238-Jmf
48/59
Step 4: Register applet as a
ControllerListener// Add ourselves as listener for player's events
player.addControllerListener(this);
} catch (MalformedURLException e) {
Fatal("Invalid media file URL!");
} catch (IOException e) {
Fatal("IO exception creating player for " +
url);
-
8/8/2019 7133238-Jmf
49/59
Controlling the Player Starting the Playerpublic void start() {
if (player != null)
player.realize(); Stopping the Playerpublic void stop() {
if (player != null) {
player.stop();
player.deallocate();
}
}
-
8/8/2019 7133238-Jmf
50/59
Responding to media events
Need to Implement ControllerListener
When the Player is realized
Posts a RealizeCompleteEvent Get the Visual component
if (( visualComponent =
player.getVisualComponent())!= null) {
cPanel.add(visualComponent);
Get the Control Panel component
When the media has reached the end Posts an EndOfMediaEvent
Rewind and start over
player.setMediaTime(new Time(0));
-
8/8/2019 7133238-Jmf
51/59
Extensibility Can extend JMF functionality in two ways:
Using Plug-ins
Effectively implementing custom processing components that can
be interchanged with standard components used by a Processor
By direct implementation
i.e implementing directly the Controller, Player , Processor,
DataSource, or DataSink interfaces
e.g. implementing a player to utilise a h/w MPEG decoder
e.g. integrating existing media engines
-
8/8/2019 7133238-Jmf
52/59
Interface Plugin
The base interface for JMF plug-ins.
A PlugIn is a media processing unit that accepts
data in a particular format and processes orpresents the data. Registered through the PlugInManager.
Methods
Open(), Close(), getName(), reset() Sub-interfaces
Codec, effect, demultiplexer , multiplexer, etc.
-
8/8/2019 7133238-Jmf
53/59
Codecs
One input and one output
Methods setInputFormat()
setOutputFormat()
getSupportedInputFormats()
getSupportedOutputFormats()
process()
input buffer, output buffer
-
8/8/2019 7133238-Jmf
54/59
Example 2
Accessing individual frames
-
8/8/2019 7133238-Jmf
55/59
FrameAccess
Problem
How to access individual decoded video frames
from a Processor while processing the media.This could be used for scanning the decoded data;
computing statistics for each video frame, etc.
-
8/8/2019 7133238-Jmf
56/59
FrameAccess Solution
use a pass-through plugin codec as a callback whenindividual frames are being processed.
Steps: 1) Build the pass-through codec.
2) Create a processor from the input file. This processorwill be used as a player to playback the media.
3) Get the TrackControls from the processor.
4) Set your codec on the video track:
TrackControl.setCodecChain(your_codec[])
-
8/8/2019 7133238-Jmf
57/59
Basic code// Get Video track as a track control
TrackControlvideoTrack = null;
for (int i = 0; i < tc.length; i++) {
if (tc[i].getFormat() instanceof VideoFormat) {videoTrack = tc[i]; break;
}
}
// Instantiate & set frame access codec to data flow
path.Codec codec[] = { new PreAccessCodec(),
new PostAccessCodec() };
-
8/8/2019 7133238-Jmf
58/59
PreAccessCodecpublic class PreAccessCodec implements Codec {
void accessFrame(Buffer frame) {
long t = (long)(frame.getTimeStamp()/10000000f);
System.err.println("Pre: frame #: " +
frame.getSequenceNumber () +
", time: " + ((float)t)/100f +
", len: " + frame.getLength());
}
Other methods, e.g. getSupportedInputFormats etc.
-
8/8/2019 7133238-Jmf
59/59
The Codecs Process method
public int process(Buffer in, Buffer out) {
// This is the "Callback" to access individual frames.
accessFrame(in);
// Swap the data between the input & output.Object data = in.getData ();
in.setData(out.getData ());
out.setData(data );
// Copy the input attributes to the output
out.setFormat(in.getFormat());out.setLength(in.getLength());
out.setOffset(in.getOffset());
top related