android's multimedia framework
TRANSCRIPT
2
These slides are made available to you under a Creative Commons Share-Alike 3.0 license. The full terms of this license are here: https://creativecommons.org/licenses/by-sa/3.0/
Attribution requirements and misc., PLEASE READ:● This slide must remain as-is in this specific location (slide #2), everything
else you are free to change; including the logo :-)● Use of figures in other documents must feature the below “Originals at”
URL immediately under that figure and the below copyright notice where appropriate.
● You are free to fill in the “Delivered and/or customized by” space on the right as you see fit.
● You are FORBIDEN from using the default “About” slide as-is or any of its contents.
● You are FORBIDEN from using any content provided by 3rd parties without the EXPLICIT consent from those parties.
(C) Copyright 2013-2014, Opersys inc.
These slides created by: Karim Yaghmour
Originals at: www.opersys.com/community/docs
Delivered and/or customized by
3
About
● Author of:
● Introduced Linux Trace Toolkit in 1999● Originated Adeos and relayfs (kernel/relay.c)● Training, Custom Dev, Consulting, ...
4
Agenda
● Architecture Basics● Display Core● Multimedia Architecture● App Dev Model● Kernel Drivers● HAL definitions● HAL modules● System Services● Stagefright● Walkthrough● Relevant Apps● References and Pointers
5
1. Architecture Basics
● Hardware used to run Android● AOSP● Binder● System Services● HAL
6
7
8
9
10
11
/frameworks/base/services/java/...
/frameworks/base/services/jni/
/hardware/libhardware/
/device/[MANUF.]/[DEVICE]/sdk/emulator/
Kernel or module
/frameworks/base/core/...
AOSP-providedASL
Manuf.-providedManuf. license
Manuf.-providedGPL-license
12
2. Display Core
● Display Hardware● Display stack in Android● Kernel driver● HAL definition● HAL module● Surface Flinger● Window Manager
13
2.1. Display Hardware
MMU
IOMMU
14
2.2. Display stack in Android
15
16
From source.android.com - CC-BY 2.5
17
From source.android.com - CC-BY 2.5
18
2.3. Kernel driver● Video memory management● Mode setting● Checking of parameters● Motorola Xoom:
● /dev/nvhdcp1● /dev/nvhost-ctrl● /dev/nvhost-display● /dev/nvhost-dsi● /dev/nvhost-gr2d● /dev/nvhost-gr3d● /dev/nvhost-isp● /dev/nvhost-mpe● /dev/nvhost-vi● /dev/nvmap● /dev/tegra-crypto● /dev/tegra_avp● /dev/tegra_camera● /dev/tegra_fuse● /dev/tegra_rpc● /dev/tegra_sema
● ... whatever hides in hwcomposer HAL module
19
2.3. HAL Definition
● /hardware/libhardware/include/hardware/hwcomposer.h
● struct hwc_procs:● invalidate()● vsync()
● struct hwc_composer_device:● prepare()● set()● dump()● registerProcs()● query()● *()
20
2.5. HAL module
● Skeleton /hardware/libhardware/modules/hwcomposer.cpp● /system/lib/hw/hwcomposer.BOARD.so● /system/lib/hw/gralloc.BOARD.so● Ex. - Mot Xoom:
● hwcomposer.tegra.so● gralloc.tegra.so
● Surface Flinger hook:● /frameworks/native/services/surfaceflinger/DisplayHardware
– HWComposer.cpp– Provides fake vsync if none is provided in HW
21
2.6. Surface Flinger
● Actual server:● /frameworks/native/services/surfaceflinger
● Client side:● /frameworks/native/libs/gui
● Client / Server interface:● ISurfaceComposerClient.cpp● ISurfaceComposer.cpp
● This is NOT an aidl'ed service● All communication is manually
marshalled/unmarshalled
22
2.7. Window Manager
● Server side:● /frameworks/base/services/java/com/android/server/wm/
– WindowManagerService.java– Session.java
● Client side:● /frameworks/base/core/java/android/view/
– WindowManager.java– WindowManagerImpl.java– ViewRootImpl.java
● Interfaces:● IWindowManager.aidl● IWindowSession.aidl
● Parameters (incl. z-order):● See WindowManager.java
23
3. Multimedia Architecture
● OpenMAX● Media Playback● Audio● Camera
24
3.1. OpenMAX
From Khronos website
25
3.2. Media Playback
From source.android.com - CC-BY 2.5
26
3.3. Audio
From source.android.com - CC-BY 2.5
27
3.4. Camera
From source.android.com - CC-BY 2.5
28
From source.android.com - CC-BY 2.5
29
From source.android.com - CC-BY 2.5
30
4. App Dev Model
● Media Playback● Audio Record● Camera
31
4.1. Media Playback
● developer.android.com sample:Uri myUri = ....; // initialize Uri here
MediaPlayer mediaPlayer = new MediaPlayer();
mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
mediaPlayer.setDataSource(getApplicationContext(), myUri);
mediaPlayer.prepare();
mediaPlayer.start();
32
From developer.android.com - ASL
33
4.2. Audio Record
● developer.android.com steps:● Create a new instance of android.media.MediaRecorder.● Set the audio source using MediaRecorder.setAudioSource(). You will probably
want to use MediaRecorder.AudioSource.MIC.● Set output file format using MediaRecorder.setOutputFormat().● Set output file name using MediaRecorder.setOutputFile().● Set the audio encoder using MediaRecorder.setAudioEncoder().● Call MediaRecorder.prepare() on the MediaRecorder instance.● To start audio capture, call MediaRecorder.start().● To stop audio capture, call MediaRecorder.stop().● When you are done with the MediaRecorder instance, call
MediaRecorder.release() on it. Calling MediaRecorder.release() is always recommended to free the resource immediately.
34
From developer.android.com - ASL
35
5. Kernel drivers
● Possibly V4Linux● Possibly ALSA● ... whatever hides in multimedia HAL modules
36
6. HAL Definition
● /hardware/libhardware/include/hardware/:
● audio_effect.h● audio.h● audio_policy.h● camera.h● camera2.h● camera3.h● camera_common.h
37
7. HAL module
● Audio skeletons in /hardware/libhardware/modules/audio● USB audio in /hardware/libhardware/modules/usbaudio● Default implementation in /hardware/libhardware/modules/camera● /system/lib/hw/:
● audio.a2dp.default.so● audio_policy.default.so● audio_policy.mpq8064.so● audio_policy.msm8960.so● audio.primary.default.so● audio.primary.mpq8064.so● audio.primary.msm8960.so● audio.usb.default.so● camera.msm8960.so
38
8. System Services
● Player● Audio Flinger● Audio Policy● Camera
39
8.1. Player
● Actual server:● frameworks/av/media/libmediaplayerservice
● Client side:● frameworks/av/media/libmedia
● Client / Server interface:● IMediaPlayerClient.cpp● IMediaPlayer.cpp● IMediaPlayerService.cpp● IMediaRecorderClient.cpp● IMediaRecorder.cpp
40
8.2. Audio Flinger
● Actual Server:● frameworks/av/services/audioflinger
● Client side:● frameworks/av/media/libmedia
● Client / Server interface:● IAudioFlingerClient.cpp● IAudioFlinger.cpp● IAudioRecord.cpp● iAudioTrack.cpp
41
8.3. Audio Policy
● Actual Server:● frameworks/av/services/audioflinger
● Client side:● frameworks/av/media/libmedia
● Client / Server interface:● IAudioPolicyService.cpp
42
8.4. Camera
● Actual Server:● frameworks/av/services/camera/libcameraservice
● Client Side:● frameworks/av/camera
● Client / Server interface:● ICameraClient.cpp● ICamera.cpp● ICameraRecordingProxy.cpp● ICameraRecordingProxyListener.cpp● ICameraService.cpp● ICameraServiceListener.cpp● IProCameraCallbacks.cpp● IProCameraUser.cpp
43
8.5. Wrapper process
● All media system services run within “mediaserver” system process.
● Started by init● Sources:
● frameworks/av/media/mediaserver
44
9. Stagefright
● Multimedia framework● Houses AwesomePlayer● Location:
● frameworks/av/media/libstagefright
● Support loadable HW codecs:● libstagefrighthw.so
45
10. Walkthrough
46
10. Relevant Apps
● Music● MusicFX● Gallery2● SpeechRecorder● VideoEditor
47
11. References and Pointers● Google's “official” doumentation:
source.android.com/devices/media.html
source.android.com/devices/audio.html
source.android.com/devices/camera/camera.html
● “Use the source, Luke”● Developer doc:
● https://developer.android.com/guide/topics/media/mediaplayer.html● https://developer.android.com/reference/android/media/MediaPlayer.html● https://developer.android.com/guide/topics/media/audio-capture.html
● “Android Multimedia Framework”, picker - slideshare● “The Android media framework”, Bert Van Dam & Poornachandra Kallare, Android Builders Summit
2014● “Android’s Stagefright Media Player Architecture”, Ivo Krka,
http://quandarypeak.com/2013/08/androids-stagefright-media-player-architecture/● “Mastering the Android Media Framework”, Google I/O 2009● https://www.khronos.org/openmax/
49
Acknowledgements:● Some figures and snippets taken from Google's
Android documentation at developer.android.com and source.android.com distributed under the Apache 2.0 license and/or CC-BY-SA 2.5.