![Page 1: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/1.jpg)
Chapter 4
Hardware & Software
That
Enables Multimedia
![Page 2: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/2.jpg)
Hardware That Enables
Multimedia
Hardware needs to be considered when you are
specifying, designing, creating, and
maintaining a multimedia presentation
Why?
![Page 3: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/3.jpg)
Display Adapters
Resolution: pixel by pixel size that defines the size of the display area limited by hardware
E.g., my laptop: max resolution 1600 x 1200 pixels
Video memory
Used to hold RGB (or CYM) pixel intensity information
Motion video
Frame rate indicates how many times per second video memory must be updated
Amount of memory updated depends on
Size of video image
Color resolution
Frame rate
![Page 4: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/4.jpg)
Image and Text Capture: OCR,
Digital Cameras, Scanners
Real image, digital image, still image, motion image
OCR: Optical Character Recognition
A method to capture printed text, as characters
Depends on
OCR software
e.g., range of fonts recognized
Quality of page being scanned
Likely will require proof reading
Still image capture: digital camera, scanner
Issues: Optics quality, precision of photocells (e.g., CCDs), dpi: dots per inch resolution
![Page 5: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/5.jpg)
Speakers/Sound
Sound formats
.WAV, .MP3
MIDI (musical instrument digital interface)
![Page 6: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/6.jpg)
Microphone
Microphone
Generates analog signal for sound
Can digitize with an A/D converter (sound input board)
![Page 7: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/7.jpg)
Video
Video Cameras, frame grabbers
Analog video:
Generates analog video signal (e.g., NTSC, PAL)
Need a frame grabber (video input card)
Converts an analog video signal to a digital image frame
Digital video:
Directly Digitizes camera image at some frame rate
E.g., digital camcorder with firewire interface
![Page 8: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/8.jpg)
Graphics Card and Graphical Processing
Unit (GPU)
![Page 9: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/9.jpg)
Positional Information
Mouse, joystick, digitizing tablets, track pads, touch screens
Positional information is analog
E.g., rotations of the wheels on a mouse device
E.g., coordinate information on a digitizing tablet; stylus pressure
This information is digitized
Change information is often important
Future developments
Eye tracking, eye gaze
In combination with mouse or other devices
![Page 10: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/10.jpg)
Mass Storage
DVD, CD-R, CD-RW, Mini disc (MD)
Direct access devices
Capacity
CD-ROM: about 720MB
MD: 74 minutes audio @ 44.1 kHz
DVD: from 4.4 GB to 16GB (8 hrs video)
Speed: Write speed, read speed, rewrite speed
Write once (CD-R), write multiple times (CD-RW; MD)
Laser disks: Analog format only?
60 minutes of 30 fps video
![Page 11: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/11.jpg)
Virtual Reality (VR)
Goal: “to provide the user with a computer-controlled environment that is as close to physical reality as possible”
VR system equipped with sophisticated output devices
And specialized input devices
Allow the user to react in a similar to “natural” manner
Getting the computer system to adapt (in part) to the human
As opposed to the human adapting to the computer system
Emphasis similar to that in interactive multimedia
A great deal of output from the computer to the user
Less input from the user to the computer
![Page 12: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/12.jpg)
Networks:
Modems & Network Interfaces
Modems Connection to phone, cable, ADSL, or ISDN
Network Computers connected via some physical
networking (e.g., LAN, Internet) hardware
Need network interface hardware (e.g., ethernet)
Required to interface to a network
Network bandwidth Modems: Lower bandwidth
Cable modem, ADSL, LAN connection: Higher bandwidth
Character encoding ASCII, EBCDIC, Unicode (16 bit), ISO/IEC
standards
![Page 13: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/13.jpg)
Software That Enables Multimedia
Operating systems
Development software
Delivery software
![Page 14: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/14.jpg)
Operating System Software
Device drivers
Multi-tasking, multi-threading
Memory management
Disk space management
Processor (CPU) management & scheduling
Window management
Servers (e.g., printing, email, web, telnet, ssh)
User level software
Includes programs to access and deal with lower levels of operating system (e.g., format a disk, partition a disk, change the priority of a task or process)
end user level applications to get work done
E.g., Macromedia Director, Power Point, Web browser
![Page 15: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/15.jpg)
Device Drivers
E.g., Display adapter driver, Firewire card driver, Network card driver, Video Camera Driver, Microphone driver
Devices have specific hardware
Drivers are specific to the hardware
Enable the operating system to use that specific hardware
Device drivers depend
on operating system
on the specific device
Device drivers enable programs
To be less dependent on hardware; more generic
![Page 16: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/16.jpg)
Codec’s
Compression and Decompression software for video and audio
“drivers” for the particular format used for the files
Depend on format, not on hardware
Within a particular general format (e.g., AVI or Quicktime)
E.g., with AVI, a Cinepak or Indeo codec can be used
Sorensen is available with Quicktime
Many codec’s can be used
Codec must be on development system and on the destination system
![Page 17: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/17.jpg)
Networking - 1
Networking functions are incorporated
into operating systems
Remote disk mounting
Network device (e.g., printer) access
Lower levels of network data transmission
(e.g., physical, data link layers)
Multimedia software often depends on
networks
Dynamic loading of codec & other components
Access to web-based presentations
![Page 18: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/18.jpg)
Networking - 2
Communication protocols
Physical layer protocols (ISO layer 1)
E.g., Ethernet, token ring, ATM
Higher protocol layers
TCP/IP (layers 2 & 3)
SMTP – simple mail transfer protocol (layer 6)
HTTP – hypertext transfer protocol (layer 6)
Performance objectives
Accuracy, transmission speed, error correction, user-perceived delays, overall throughput
![Page 19: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/19.jpg)
Client/Server Architecture - 1
Server
A process that provides access to particular hardware or software resources
Process: A program that runs independently on a computer
Often used in the context of networks
Used within software design generally, as well
“Server” may designate a software process
Or may designate a computer (hardware)
Depends on the kind of service being provided
![Page 20: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/20.jpg)
Client/Server Architecture - 2
E.g., database server
Computer will have database software
May also have additional hardware resources (e.g., more disk space)
E.g., a software license server (“metering”)
Enables some maximum number of people to be using a software resource at a given time.
Server receives requests
Remote requests from a client process on another computer
Local requests from a client process on same computer
Requests are sometimes for data sometimes for programs
![Page 21: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/21.jpg)
Operating System &
Multimedia Requirements
Development of multimedia
Running multimedia
Some operating systems may be more suited for running multimedia
Issues include:
Memory (RAM) required
Operating system availability
Application availability
Speed
Networking
![Page 22: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/22.jpg)
Multimedia Development
Software
Software needed to develop multimedia
And software needed to deliver
E.g., Shockwave playback in a browser has different requirements than running Macromedia Director
Development
Preproduction, production, postproduction
Types of development software
2D image capture, scanning & processing; Digital video capture & processing
Animation; 3D modeling, rendering; Sound capture & processing; Text; Web development; Authoring
![Page 23: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/23.jpg)
Vector vs. Bitmap Graphics - 1
Representation of objects in images E.g., a rectangle with a black outline and a gray
interior
Bitmap (raster) graphics Specify value for each pixel
Application data file (e.g., Photoshop) stores collection of pixels making up rectangle
E.g., 300 x 600 = 180,000 pixels of rectangle
Requires 176KB to 527KB of data (uncompressed)
Vector graphics Use rectangle drawing tool & color-fill
Application program generates a data file that represents rectangle as a command and parameters
E.g.,
Rectangle, 0, 0, 300, 600, black, gray (32)
Requires about 100 bytes of information
![Page 24: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/24.jpg)
Vector vs. Bitmap Graphics - 2
Space Tradeoff
Bitmap graphics (uncompressed) take more space
Vector graphics take less space
Speed Tradeoff
Bitmap graphics (uncompressed) are drawn faster
Vector graphics need to be decoded and then the bitmap (pixels) generated
Will be slower than bitmap representation
An issue on slower computers or if a very large number of objects need to be displayed
Expressiveness
Some objects can’t be easily drawn in vector fashion
Contemporary drawing programs may provide both vector and bitmap graphics
![Page 25: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/25.jpg)
Video - 1
Video is captured in discrete units called frames Frame: Similar to a still picture
Temporal and spatial compression Spatial: within a frame
Temporal: across frames
Key-frame concept in temporal compression Some compression methods use this form of coding
Store information on one frame at particular intervals
This is a “key frame”
Then store changes from this point
Uses frame differencing
Makes use of common background often shared between multiple frames
![Page 26: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/26.jpg)
Video - 2
Time-dependent medium
Illusion of motion depends on frame rate
Don’t perceive motion below about 15 fps
Also: animation and sound
Time-dependent media place high computational demands on a computer
E.g., with 1024 x 768 image resolution, and 24 bit color depth @ 30 fps, what is the (apparent) data transfer rate in KB/second?
![Page 27: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/27.jpg)
Video - 3
1024 * 768 = 786, 432 pixels/frame
786, 432 pixels/frame* 3 bytes/pixel =
2,359,296 bytes/frame
2,359,296 bytes/frame * 30 frame/sec =
70,778,880 bytes/sec
(70,778,880 bytes/sec) / (1024 bytes/KB) =
69,120 KB/sec
(Apparent: because this is without any data compression)
![Page 28: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/28.jpg)
Animation
Similar to video
Each frame created via software
not from a camera source
History
traditionally done by hand, frame by frame
Model-based animation
Online (real-time) versus offline
Model-based: software representation of world
Online: Frames are generated on-the-fly, in response to dynamic changes of the model
Offline: Frames generated in response to model changes
![Page 29: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/29.jpg)
Related Techniques
Biological animation
Animal gait (مشية الحيوان), bipedal motion (حركة القدمين)
Morphing (تتحول) Gradual changes in one image to transform
it to a second image
Applying changes gradually
E.g., taking a few seconds @ 30 fps
Can be used for animation purposes
VRML
Virtual reality modeling language
"an open standard for 3D multimedia and shared virtual worlds on the Internet."
![Page 30: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/30.jpg)
MPEG, Quicktime, AVI
Standard formats for video and audio
information
Includes techniques/standards for
compression, decompression
MPEG
MPEG Audio
ISO/IEC standardization
Open standard
Specification is available to everyone (for a fee)
No single company “owns” the standard
![Page 31: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/31.jpg)
Quicktime, AVI
Quicktime
Primarily Macintosh/Apple
http://www.quicktime.com
AVI
Primarily Windows/Microsoft
AVI: “Audio Video Interleave”
E.g., see http://www.jmcgowan.com/avi.html
![Page 32: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/32.jpg)
MPEG Standards
MPEG-1:
Applications: MP3, Video CD
MPEG-2
MP3, Digital Television set top boxes, Digital Versatile Discs (DVD).
MPEG-4
Proposed fixed & mobile web standard
MPEG-7: Multimedia Content Description
MPEG-21: Multimedia Framework
![Page 33: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/33.jpg)
MPEG: Video
MPEG-1
Roughly VHS quality at 1,15 Mbit/s
MPEG-2
“MPEG-2 Video was at least good, and in many cases even
better than standards or specifications developed for high
bitrate or studio applications”
http://mpeg.telecomitalialab.com/standards/
mpeg-2/mpeg-2.htm
![Page 34: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/34.jpg)
MPEG: Audio
MPEG standards have layers
Layer: Operating modes with increasing complexity and performance
MPEG-1 provides
Mono and stereo coding at 32, 44.1, and 48 kHz sampling rate
Layer I, II, III: varying bit rates from 32 to 448 kbit/s
Optimized for 128 kbits/s stero
MPEG-2 BC provides
Backwards compatible multichannel extension to MPEG-1
Up to 5 channels plus a 'low frequent enhancement' channel
Bit rate range is extended up to about 1 Mbit/s;
an extension of MPEG-1 towards lower sampling rates 16, 22.05, and 24 kHz for bitrates from 32 to 256 kbit/s (Layer I) and from 8 to 160 kbit/s (Layer II & Layer III).
http://mpeg.telecomitalialab.com/faq/audio.htm#5
![Page 35: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/35.jpg)
MPEG-2 & MPEG-4: Audio
MPEG-2 Advanced Audio Coding (AAC)
High-quality audio coding standard for 1 to 48 channels
Sampling rates of 8 to 96 kHz
Multichannel, multilingual, and multiprogram capabilities
Bit rates from 8 kbit/s to more than 160 kbit/s/channel
Multiple encode/decode cycles
Three profiles (layers) of AAC provide varying levels of complexity and scalability.
MPEG-4 will provide
coding and composition of natural and synthetic audio objects at a very wide range of bit rates.
Emphasis on new functionalities
![Page 36: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/36.jpg)
MPEG-4
Goal: Higher levels of interaction
1. represent units of natural, visual or audiovisual content, called "media objects". These media objects can be of natural or synthetic origin; this means they could be recorded with a camera or microphone, or generated with a computer;
2. describe the composition of these objects to create compound media objects that form audiovisual scenes;
3. multiplex and synchronize the data associated with media objects, so that they can be transported over network channels providing a QoS appropriate for the nature of the specific media objects; and
4. interact with the audiovisual scene generated at the receiver’s end.
http://mpeg.telecomitalialab.com/standards/mpeg-4/mpeg-4.htm
![Page 37: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/37.jpg)
MP3
MPEG-1/2 Audio Layer-3
MPEG-1 Audio Layer-3
MPEG-2 Audio Layer-3 (successor)
MP3 and AAC Explained
http://citeseer.nj.nec.com/421409.html
MP3 MPEG-3!
![Page 38: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/38.jpg)
Sound Representation
On a sample basis E.g., about 44 kHz for CD quality audio
Each sample is a number
gives the amplitude of the analog sound wave at that point
On a feature basis E.g., frequency, duration, musical note
Also possible: for speech, phonemes
Phoneme is a unit of sound in a particular language
Phonemes may sound differently dependent on context
E.g. /p/ (personal, paternal, computer, rapid), /b/, and /s/
Difference is analogous to the difference between bitmapped (raster) & vector graphics
Tradeoffs– Similar to those between bitmap & vector graphics
![Page 39: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/39.jpg)
Example 1
With a sampling rate of 44.1 kHz, and 16 bitsound resolution, how many KB does it take to represent 3 minutes of sound? (Without compression)
![Page 40: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/40.jpg)
Example 1
44,100 samples/sec * 2 bytes/sample =88, 200 bytes/sec
88, 200 bytes/sec * 60 sec/minute =
5,292,000 bytes/min
5, 292, 000 bytes/min * 3 min =
15,876,000 bytes
15,876,000 bytes/(1024 bytes/KB) =
15, 503.90625 KB
(Or about 15 MB)
![Page 41: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/41.jpg)
Example 2
If we are representing speech phonemes, and we want to represent individual words, how much data does it take to encode the English word “Hello”?
Assume that English has 39 phonemes
Assume that we have one phoneme per character
Problem is to represent sequences of phonemes
Each phoneme can take on 39 possible values
Need at least 6 bits of information to represent 64 different values , So, need at least 6 bits per phoneme
Word “Hello” has 5 characters
Assuming also 5 phonemes 5 * 6 = 30 bits
30 bits/(8 bits/byte) = 3.75 bytes Need 4 bytes
![Page 42: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/42.jpg)
Sound Filtering
A sound can be filtered to remove a certain set of frequencies
Some types of filtering
Low pass: Allows only low frequencies through
High pass: Allows only high through
Band pass: Allows only a band range of frequencies through
Other types of filtering or processing
Trimming, splicing, volume
![Page 43: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/43.jpg)
Web browsers
Process various file types
HTML
Media formats: images, sounds, movies
Some types of data are processed directly
Built into the browser
Other types
Require additional programs or “plugin-ins”
E.g., a plugin to display PDF or Adobe Portable Document format files or a plugin to display Shockwave files
Some plugins are called “players” or “viewers”
E.g., a viewer to display MS Word documents
![Page 44: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/44.jpg)
Multimedia platform
requirements:
Multimedia platform should
Support
the development of a wide variety of multimedia applications
applications in a distributed system
Be able to
establish and control streams of continuous media
to choose the quality of service for a data stream
to express relations between several media streams
![Page 45: Chapter 4 Hardware & Software That Enables Multimedia](https://reader035.vdocuments.site/reader035/viewer/2022070923/62c7f0bba8a1585dc94655db/html5/thumbnails/45.jpg)
Multimedia Delivery Software
Two general means of delivery Compiled or Interpreted
Compiled Machine code generated for a computer CPU
(e.g., Intel-based) and operating system (O/S)
Faster; limited to computer type & O/S; stand-alone
E.g., Macromedia Director “Projector”
Interpreted Instructions created that can be interpreted by
a program; E.g., a plugin
Slower; more portable; need interpreter (e.g., player) software
E.g., Macromedia Shockwave
In both cases, media files are embedded and/or retained as separate files