multimedia databases, multi-terabyte performance oracle10 g intermedia jim steiner senior director...
Post on 21-Dec-2015
226 views
TRANSCRIPT
Multimedia Databases, Multi-Terabyte Performance
Oracle10g interMedia
Jim SteinerSenior Director
Server Technologies Oracle Corporation
Session ID: 40121
Jeremy FormanComputer System Analyst
New Mexico Department of Transportation
Agenda
• Oracle’s Multimedia CapabilitiesOracle10g New FeaturesHow Oracle ComparesHow users benefitMulti-Terabyte, Multimedia databasesJeremy Forman -- New Mexico DOT
Objective
Extend Oracle’s leadership as a platform capable of managing multimedia content as naturally as it does all other business information.
Lower the cost and complexity of developing, deploying and managing business applications which make extensive use of multimedia data.
The Media-enabled Oracle Platform
Oracle Database 10g
– Storage, management, & retrieval of image, audio, video data
– Native format understanding, metadata extraction, methods for
image processing
– Support for leading streaming media servers
Oracle Application Server 10g
– JSP, servlet and PL/SQL application development support
– Media Adaptation Services for Wireless
– JDeveloper (BC4J/UIX) and Portal integration
Oracle Collaboration Suite– Metadata extraction for OCS Files
Oracle10g New Multimedia Features
Multimedia and the Grid
Applications that make extensive use of multimedia face the same challenge as most business applications
– Performance– Scalability– High level of service– At lowest possible cost
Multimedia applications often have greater storage, distribution, security, and “demand peaks” requirements
Enterprise Grid Computing benefits multimedia applications through dynamic provisioning of resources
New Oracle10g Multimedia Features
Standards Support – SQL/MM Still Image
New version of Java Advanced Imaging and
additional image processing operators
Support for additional media formats
– Microsoft ASF, MPEG2 & MPEG4
• Microsoft Windows Media Server Plugin
• Real Server Plugin for Helix Server
• XML DB integration
How Oracle’s Multimedia capabilites are better than other DBMSesOnly Oracle10g: Supports media content natively
– No manual initiation of separate processes to enable database tablespace to accept media data.
– No need for DBAs to initiate these processes for each table where they wish to store media data
Stores all media and its metadata in the same table as the associated relational data
– No triggers on each and every media object created to update the separate “administration” tables that contain media objects and metadata.
– No added processing and I/O overhead for access and retrieval Provides Java class libraries and JSP Tag libraries for
application development and media access.
How do users benefit?
Oracle Performs
Fast retrieval – 1TB image repository renders
images in Web browser in less than 0.4 second
Load at device speeds
Oracle Scales
Multi-terabyte multimedia databases– 140 million images– 5 TB database
Scalable bulk load and process– Parallel processes load 300,000
images/hour– Bulk process – tiff to gif conversion, scale
Oracle is Easier to Manage
Multimedia VLDB is easier to manage using Oracle
– RMAN for very large backup (3TB database)
– Single DBA for 5TB database
Oracle is More Secure
Banks use it Multimedia data inherit all of the built in
security features of the Oracle Database– authentication, auditing, encryption,
access control . . .– Image and media features enhance
security applications
Oracle Simplifies CodeImage Insert using Multimedia JSP Tag Library– An Example
With JSP Tag Library: (14 point font)
<ord:embedImage connCache = <% java.util.Vector otherValuesVector = new
java.util.Vector(); otherValuesVector.add(fd.getParameter("desc")); otherValuesVector.add(fd.getParameter("loc"));
%> “ mediaParameters = "photo" otherColumns = "description, location" otherValues = "<%=otherValuesVector%>" />
Without: (in 10 point font) <FORM ACTION="PhotoAlbumInsert.jsp" METHOD="POST"
ENCTYPE="MULTIPART/FORM-DATA"> Description: <INPUT TYPE="text" NAME="desc"><BR> Location: <INPUT TYPE="text" NAME="loc"><BR> Photo: <INPUT TYPE ="file" NAME="photo"><BR> <INPUT TYPE ="submit" VALUE="submit"></FORM> try { // Parse multipart/form-data formData.setServletRequest( request ); formData.parseFormData();
// Insert new row into database stmt = (OraclePreparedStatement)conn.prepareStatement( "insert into spec_photos ( description, location, photo ) " + " values ( ?, ?, ORDSYS.ORDImage.init() )" ); stmt.setString( 1, formData.getParameter( "description" ) ); stmt.setString( 2, formData.getParameter( "location" ) ); stmt.executeUpdate(); stmt.close(); // Fetch OrdImage object from database stmt =
(OraclePreparedStatement)conn.prepareStatement( "select photo from spec_photos where description = ? for update" );
stmt.setString( 1, formData.getParameter( "description" ) ); rset = (OracleResultSet)stmt.executeQuery(); rset.next(); OrdImage photo = (OrdImage)rset.getCustomDatum( 1, OrdImage.getFactory()); rset.close(); stmt.close(); // Load the photo into the database and set the properties. formData.getFileParameter( "photo" ).loadImage( photo ); // Update object in database stmt = (OraclePreparedStatement)conn.prepareStatement( "update spec_photos set photo = ? where description = ?" ); stmt.setCustomDatum( 1, photo ); stmt.setString( 2, formData.getParameter( "description" ) ); stmt.execute(); stmt.close();
// Commit changes conn.commit(); } finally { // Ensure JDBC connection is released and any temp files are deleted. album.release(); formData.release(); }%>
Saves Money“Central Bank” in nearby west coast city:
• Reduces the administrative cost and the cost of float for member banks.
• Enables on-line processing and rapid resolution of 26,000 bad checks each day.
• Electronic transmission of check images and management in Oracle9i Database
Saves TimeCaixa Economica Federal - Largest Brazilian Government bank:• interMedia automatically transforms original files .tiff to .gif during the load process – buying time
and saving money.• Direct access by users to statement information with in secure fashion
Palazzo Braschi Museum - Rome:• Reduced the time to process images by 90% using interMedia methods to bulk load and process
image content compared to using client side tools. • Processing includes format conversion, thumbnail generation, metadata extraction, index & load
Saves Labor
New Mexico Department of Transportation:
• A single DBA designed, created, deployed, and maintains a 5 TB image management system
Multi-Terabyte, Multimedia databases
Best Practices for Multi-Terabyte, Multimedia databases
Storage planning: Media data may grow at faster rates from other data and will require different management strategies.
– Store media data in separate tablespaces that can be tuned to meet these needs.
– Consider the use of partitioning to spread media data over more I/O storage devices.
LOB tuning – Use a large CHUNK value (32K maximum) to
specify LOB storage. This increases I/O efficiency. Only time you would not want to specify max value is when most media is smaller than 32K.
Best Practices for Multi-Terabyte, Multimedia databases Loading media:
– Initialize interMedia objects with empty LOB locators (Use the init() constructors). This allocates the required space in row on the database block when the row is inserted.
– Consider disabling LOB LOGGING for media segments. This saves the cost of writing the media to the Redo log as well as to the tablespace.
– If LOGGING is enabled, increase the LOB_BUFFER parameter to allocate more space for the extra media data that is logged. Consider using larger size redo log files to decrease the number of log switches.
– Parallelize your loading so that media is written to multiple I/O storage devices. Avoid bottlenecking your load on a single storage device.
Best Practices for Multi-Terabyte, Multimedia databases
Retrieving media:– Use the CACHE option on LOBs if the same LOB
data is to be accessed repeatedly.– Increase the DB_CACHE_SIZE parameter to account
for increased cache requirements of media data.
Processing image data:– Many image processing operations will fully
decompress a compressed image in order to perform the operation. Decompressed images can be ten times the size of a compressed image.
– Increase the JAVA_POOL_SIZE parameter to allocate enough memory to process these decompressed images.
Multimedia DatabasesMultiterabyte Customer Examples
Financial– Caixa Economica, Brazil: 4TB bank statement image database– US ‘Central’ Bank: 1TB check image database– UBS Paine Webber: 1TB check image database
Healthcare– Michigan – Medical records repository – long term care assessment
Education– University of Oslo, Norway – 1 TB National Museum digital repository– Online Computer Library Center, Inc (OCLC) – 5TB+ Digital Library
Government– US Navy – Award winning LIFELines Portal also w/Oracle Portal– State of New Mexico D.O.T. – 5 TB image database w/Oracle Portal
Summary
Oracle treats multimedia like any other data
Users Save Money, Labor, and Time
But enough talk – lets see a real application
deployed by the State of New Mexico . . .
Jeremy FormanComputer System Analyst
New Mexico Department of Transportation
Some Useful New Oracle 10g Features
Support for Partitioning of tables with object columns in tablespaces with Automatic Space Management
Data Pump Import and Export 4 GB RAM Tuning (4GT) on Windows (beta not
available for 64 bit)
4GT On Windows -- Performance
Allow Memory-intensive applications running on Oracle10g Enterprise Edition to access up to 3 GB of memory
50 percent more memory is available for database use, increasing SGA sizes or connection counts
The Road Features Inventory
Multiterabyte database (4TB) Approximately 5,000,000 images 1,000,000 Assets Web based Application
D E M O N S T R A T I O N
The Road Features Inventory
Oracle interMedia
Media and application metadata management services
Storage and retrieval services Support for popular formats Access through traditional and Web interfaces
and a search capability using associated relational data or using specialized indexing
Oracle interMedia supports multimedia storage, retrieval, and management of:
Binary large objects (BLOBs) stored locally in Oracle10g and containing audio, image, or video data
File-based large objects, or BFILEs, stored locally, containing audio, image, or video data, or other heterogeneous media data
URLs containing audio, image, or video data or other heterogeneous media data, stored on any HTTP server such as Oracle Internet Application Server
Streaming audio or video data stored on specialized media
interMedia Object Types
ORDAudio ORDDoc ORDImage ORDVideo ORDImageSignature
ORDImage Attributes
source: the source of the stored image data. height: the height of the image in pixels. width: the width of the image in pixels. contentLength: the size of the on-disk image file in bytes. fileFormat: the file type or format in which the image data is
stored (TIFF, JIFF, and so forth.). contentFormat: the type of image (monochrome and so
forth). compressionFormat: the compression algorithm used on
the image data. mimeType: the MIME type information.
ORDImage Methods
Init() ProcessCopy() Set/GetUpdateTime() Set/GetMimeType() GetCompressionFormat() ReadFromSource() WriteToSource()
interMedia Image Loading
PL/SQL SQLLDR Java
SQLLDR Example
LOAD DATA INFILE * INTO TABLE SOUNDS APPEND FIELDS TERMINATED BY ',' (Item_ID integer external, sound column object ( source column object ( localdata_fname FILLER CHAR(128), localdata LOBFILE (sound.source.localdata_fname), ) ) )
BEGINDATA 55,the_grid.au, 33,engine.wav, 44,spacemusic.au
INSERT INTO stockphotos VALUES ( 1, 'John Doe', 'red plaid', ORDSYS.ORDImage.init(), ORDSYS.ORDImageSignature.init());
Inserting an Image
interMedia: Loading ImagesSELECT RIMAGES_VIRTDRIVE_SEQ.NEXTVAL INTO v_NextSeqVal FROM DUAL;
INSERT INTO RFI_VIRTUAL_DRIVE_IMAGES(ID,RROUTES_ID,ROUTE_PREFIX,ROUTE_ID,
DIRECTION,SMPOINT,FILENAME,IMAGE,THUMBNAIL)
VALUES (v_NextSeqVal, p_rroutes_id, p_RoutePrefix, v_RouteID,
v_Direction, v_Smpoint, p_ImageName,
ORDSYS.ORDImage.init(), ORDSYS.ORDImage.init()), ORDSYS.ORDImageSignature.init());
-- Select the newly inserted row for update
SELECT IMAGE INTO v_Image FROM RFI_VIRTUAL_DRIVE_IMAGES
WHERE ID = v_NextSeqVal FOR UPDATE;
-- This procedure imports the image file from the RFI_IMAGES directory on a the local file system
-- (srcType=FILE) and automatically sets the properties.
v_Image.setSource('file','FINAL_JPEGS', p_ImageDir||'\'||p_ImageName);
v_Image.import(ctx);
UPDATE RFI_VIRTUAL_DRIVE_IMAGES SET IMAGE = v_Image WHERE ID = v_NextSeqVal;
--Call the Copy to Thumbnail Procedure
Rfi_Load_Images_Final.Copy_To_Thumbnail(v_NextSeqVal);
interMedia: Copying an Image
BEGIN
SELECT IMAGE INTO v_Image_1
FROM RFI_VIRTUAL_DRIVE_IMAGES WHERE ID = p_ImageID;
SELECT THUMBNAIL INTO v_Image_2
FROM RFI_VIRTUAL_DRIVE_IMAGES WHERE ID = p_ImageID FOR UPDATE;
-- Convert the image to a TIFF thumbnail image and store the
-- result in Image_2
v_Image_1.processcopy('maxScale=250,250', v_Image_2);
-- Continue processing
UPDATE RFI_VIRTUAL_DRIVE_IMAGES SET THUMBNAIL = v_Image_2
WHERE ID = p_ImageID;
COMMIT;
Dynamically Changing Image Format -- Some image formats are supported by interMedia but may not be able
-- to be displayed in-line by a browser. The BMP format is one example.
-- Convert the image to a GIF or JPEG based on number of colors in the
-- image.
IF new_image.contentFormat IS NOT NULL AND ( new_image.mimeType = 'image/bmp' OR new_image.mimeType = 'image/x-bmp' ) THEN BEGIN new_image.process( 'fileFormat=' || get_preferred_format( new_image.contentFormat ) ); EXCEPTION WHEN OTHERS THEN NULL; END; END IF;
FUNCTION get_preferred_format( format IN VARCHAR2 ) RETURN VARCHAR2
IS
num_digits INTEGER;
ch CHAR(1);
BEGIN
-- Image content format strings have the following format: <#bits><format> MONOCHROME
-- Figure out the number of digits that represent the number of colors.
num_digits := 0;
LOOP
ch := SUBSTR( format, num_digits + 1, 1 );
IF ch >= '0' AND ch <= '9‘ THEN
num_digits := num_digits + 1;
ELSE
EXIT;
END IF;
END LOOP;
Dynamically Changing Image Format
-- Images with more than 8 bits of color can be converted to the JPEG-- format without significant discernible loss of quality.
IF num_digits > 0 THEN IF TO_NUMBER( SUBSTR( format, 1, num_digits ) ) > 8 THEN RETURN 'JFIF'; END IF; END IF; -- Images with 8 bits of color or less are best converted to the GIF format to retain the quality. RETURN 'GIFF';END get_preferred_format;
Dynamically Changing Image Format
Content Based Retrieval with ORDImage
•The primary benefit of using content-based retrieval is reduced time and effort required to obtain image-based information
•A content-based retrieval system processes the information contained in image data and creates an abstraction of its content in terms of visual attributes
Our Multiterabyte Environment
The Server
Windows 2000 Advanced Server Compaq Proliant w/ 8GB RAM 4 - 700Mhz Processors ¾ TB Local Storage 4 TB IBM Shark Storage 24 different physical drives 200GB per drive
The Database
120 Tablespaces Average Datafile: 16GB Materialized Views Partitioning for most tables, materialized
views, and Indexes
Partitioning and a Multiterabyte Database
Partitioning allows tables, indexes, and index-organized tables to be subdivided into smaller pieces, enabling these database objects to be managed and accessed at a finer level of granularity.
Partitioning
List Partitions Range Partitions Hash Partitions Composite Range-Hash Partitioning Composite Range-List Partitioning Sub Partitions
List Partitioning Example
CREATE TABLE q1_sales_by_region (deptno number, deptname varchar2(20), quarterly_sales number(10, 2), state varchar2(2)) PARTITION BY LIST (state) (PARTITION q1_northwest VALUES ('OR', 'WA'), PARTITION q1_southwest VALUES ('AZ', 'UT', 'NM'), PARTITION q1_northeast VALUES ('NY', 'VM', 'NJ'), PARTITION q1_southeast VALUES ('FL', 'GA'), PARTITION q1_northcentral VALUES ('SD', 'WI'), PARTITION q1_southcentral VALUES ('OK', 'TX'));
Querying Directly from a Partition
SELECT DEPTNO, DNAME
FROM Q1_SALES_BY_REGION PARTITION(Q1_SOUTHWEST)
WHERE STATE = ‘NM’
CREATE TABLESPACE <interMedia Tablespace> DATAFILE <datafile_name>BLOCKSIZE 16384EXTENT MANAGEMENT LOCALSEGMENT SPACE MANAGEMENT MANUAL
Oracle 10g supports partitioning objects in Tablespaces with Automatic Segment Space Management
Partitioning and interMedia
Partitioning an interMedia Table
CREATE TABLE RFI_VIRTUAL_DRIVE_IMAGES ( ID NUMBER (9) NOT NULL, RROUTES_ID NUMBER (5), FILENAME VARCHAR2 (100) NOT NULL, SMPOINT NUMBER (11,3) NOT NULL, DIRECTION NUMBER (1), IMAGE ORDIMAGE, THUMBNAIL ORDIMAGE, SIGNATURE ORDIMAGE, ROUTE_PREFIX VARCHAR2 (2), ROUTE_ID NUMBER (6), CREATEDBY VARCHAR2 (30), CREATEDDATE DATE, MODBY VARCHAR2 (30), MODDATE DATE)
Partitioning an interMedia Table
PARTITION BY RANGE(ID)SUBPARTITION BY HASH(ROUTE_ID)SUBPARTITIONS 12(PARTITION IMAGES_PART1 VALUES LESS THAN(500000) TABLESPACE RFI_PART1_DIST1_TBSLOB (IMAGE.SOURCE.LOCALDATA) STORE AS(TABLESPACE RFI_IMAGES_PART1_TBS NOCACHE CHUNK 16346),LOB (THUMBNAIL.SOURCE.LOCALDATA) STORE AS(TABLESPACE RFI_THUMBNAILS_PART1_TBS NOCACHE CHUNK 8132) LOB (SIGNATURE.SOURCE.LOCALDATA) STORE AS(TABLESPACE RFI_SIGNATURES_PART1_TBS NOCACHE CHUNK 8132)
Partitioning an interMedia Table
PARTITION IMAGES_PART2 VALUES LESS THAN (MAXVALUE) TABLESPACE RFI_PART2_DIST2_TBSLOB (IMAGE.SOURCE.LOCALDATA) STORE AS(TABLESPACE RFI_IMAGES_PART2_TBS NOCACHE CHUNK 16346) LOB (THUMBNAIL.SOURCE.LOCALDATA) STORE AS(TABLESPACE RFI_THUMBNAILS_PART2_TBS NOCACHE CHUNK 8132) LOB (THUMBNAIL.SOURCE.LOCALDATA) STORE AS(TABLESPACE RFI_THUMBNAILS_PART2_TBS NOCACHE CHUNK 8132)
interMedia Code Wizard
Oracle8i: download the Code Wizard for the PL/SQL Gateway:
– http://otn.oracle.com/software/products/intermedia/content.html
Oracle9i: <oracle_home>/ord/http/demo/plsgwycw Oracle10g: <oracle_home>/ord/http/demo/plsgwycw Execute ordplsui.sql as ORDSYS Create Code Wizard administration DAD Grant Access to other DAD
Next Steps…. Recommended sessions
– BioMed Central– JDeveloper Sessions
Recommended demos and/or hands-on labs– JDeveloper Campground– Server Technologies Campground – interMedia Demo– Portal Demo
See Your Business in Our Software– Visit the DEMOgrounds for a customized architectural review, see a
customized demo with Solutions Factory, or receive a personalized proposal. Visit the DEMOgrounds for more information.
Relevant web sites to visit for more information– http://otn.oracle.com/products/intermedia/
interMedia User’s GuideinterMedia Reference
AQ&Q U E S T I O N SQ U E S T I O N S
A N S W E R SA N S W E R S