berkeley db: data storage for the edge and beyond · 31 sql support a peek inside the box •...

<Insert Picture Here>

Berkeley DB: Data Storage for the Edge and Beyond

Ashok Joshi

Senior Director, Oracle Berkeley DB and Database Mobile Server

development, Oracle NoSQL Database development

2Copyright © 2012, Oracle Corporation Japan. All rights reserved.

以下の事項は、弊社の一般的な製品の方向性に関する概要を説明するもので

す。また、情報提供を唯一の目的とするものであり、いかなる契約にも組み

込むことはできません。以下の事項は、マテリアルやコード、機能を提供す

ることをコミットメント（確約）するものではないため、購買決定を行う際

の判断材料になさらないで下さい。オラクル製品に関して記載されている機

能の開発、リリースおよび時期については、弊社の裁量により決定されます

。

OracleとJavaは、Oracle Corporation 及びその子会社、関連会社の米国及びその他の国における登録商標です。

文中の社名、商品名等は各社の商標または登録商標である場合があります。


Embeddable database libraries

• Berkeley DB is NOT a server

• Embed Berkeley DB within your application to

get

• Transactions

• Recovery

• Scalability

• High availability

• Indexing

• Concurrency and high performance

• Every application needs to manage data


Build vs. Buy

• Consider rebuilding following components

• Graphic Engine

• Network Protocol Stack

• Encryption Library


Build vs. Buy

• Consider rebuilding following components

• Graphics Engine

• Network Protocol Stack

• Encryption Library

• Data Storage?

CoreFunctions

GraphicStorage

Network Encrypt


Benefits of Berkeley DB vs. building

your own solution

• Time to market

• Design, development time and cost

• Resources

• Focus on company‟s core competencies

• Quality

• Proven reliability with various situations

• Lifecycle

• „Software maintenance‟ is defined as the following (IEEE‟91)

• Perfective maintenance

• Adaptive maintenance

• Corrective maintenance

• Preventative maintenance


Benefits of Berkeley DB …

• Commercial solution from leading vendor

• Leverages success of Relational DBMS, SQL, XML/XQuery DB

• Addresses the need for customizable data management

• Transaction

• Recovery

• Replication

• Scalability

• Character Set

• LRU data caching

• Concurrency/Locking

• Indexed access

• SQL

• XML schema and query


Berkeley DB Value Proposition

• Small footprint, efficient and lightweight

• Industrial grade performance, reliability and availability

• Highly flexible, configurable, easy to use

• Maintenance-free deployment


History of Berkeley DB

• ‟91 Started from BSD Unix

• AT&T‟s „dbm‟ replaced by „bdb‟

• Single user (no locking), no transactions, no recovery, few tools

• ‟96 Netscape‟s requirements

• Locking, transactions, ACID compliance, tools, …

• Without installation and no DBA!

• ‟96 Sleepycat founded

• For commercial support

• ‟06 Acquired by Oracle


Did you know Berkeley DB...?

• Is everywhere

• runs on 200 million machines across the world

• Scales up, scales down

• powers mobile phones to stock exchanges

• Is proven

• runs inside every copy of Linux (and MacOS X and Solaris)

• Powers the Internet

• used extensively by Akamai, Yahoo, AOL, Amazon, eBay and many others

• Is highly available

• 70% of the global daily email traffic goes through BDB


Berkeley DB Product Overview

• Embeddable database technology

• Small footprint, efficient and lightweight

• Fast, scalable, transactional, recoverable

• Industrial grade performance, reliability and availability

• Highly flexible, portable, configurable, easy to use

• Easily and seamlessly integrated into any application (C, C++, C#,

Java, XML, scripting languages, ...)

• Drop-in replacement/upgrade for SQLite

• Manage and synchronize mobile and remote data with the Oracle DB

• Proven in 200+ million deployments

• Used by all versions of Linux, BSD Unix, OpenLDAP, OpenOffice and

hundreds of other open source and commercial software packages

BDB in Devices and Appliances

Key Benefits• Small Footprint

• Synchronization

• Performance

• Scalability

• Reliability

• Embeddable

Storage

Smart Metering,

Smart Grid, M2M

Medical &

Life Science

Devices

Mobile Applications

Email &

Messaging

Entertainment &

Gaming Devices

BDB in Infrastructure

Key Benefits• Performance

• Scalability

• Reliability

• High Availability

• Embeddable

NetworkingEquipment

Authentication /Identity Management

Gateways

SOA / WebServices /

EAI

Caching /Logging

Search /Content Management

Resource /

Systems Mgmt


Understanding

BerkeleyDB APIs


RDBMS vs. Berkeley DB

Your Application Code

ODBC, JDBC, OCI, etc.

TCP/IP

TCP/IP

SQL Parser

Query OptimizerYour Application Code

Berkeley DBStorage Manager

Conventional RDBMS Berkeley DB


Key/value pairs

000001 “slkdjflskdjflskdfj”000001 “slkdjflskdjflskdfj”000001 “slkdjflskdjflskdfj”“000001” “가나다라”

Key Value


Key/value vs. Relational

ID First name, Last name, Phone number, City, State, Zip

key value

First name Last name Phone Number City State ZipID First name Last name Phone Number City State ZipID ＋ First name Last name Phone Number City State ZipID

ID First name, Last name, Phone number, City, State, ZipID First name, Last name, Phone number, City, State, Zip

Primary Key

Columns in Relational Model

Key/value of Berkeley DB


ID First name, Last name, Phone number, City, State, Zip

key value

Index

First name Last name Phone Number City State ZipID First name Last name Phone Number City State ZipID ＋ First name Last name Phone Number City State ZipID

ID First name, Last name, Phone number, City, State, ZipID First name, Last name, Phone number, City, State, Zip

Primary Key

Columns in Relational Model

Key/value of Berkeley DB

IndexPhone Number

Phone NumberPhone Number

IDID

ID

Phone number IDSecondary DB

Callback function


Cursor

005 abcdedf

Key Value

002 abcdef…

001 abcd…

013 dkdkw

005 dbaaa

093 dkdkw

……


Btree

Access Methods

Hash

Queue & Recno

Heap


Compression

• Btree supports compression

• Prefix compression is default behavior

• Custom compression algorithms

ABD010320DBCEF1

ABD010320DBCEF2

ABD010320DBCEF3

ABD010320DBC3A5

ABD010320DBCEF1

ABD010320DBCEF2

ABD010320DBCEF3

ABD010320DBC3A5


Encryption

• Rijndael/AES for encryption/decryption

• Configured using 128bit Key

• 16-byte initialization vector generated using Mersenne Twister

• Checksum using SHA1, using 160-bit message digest

• Encrypts DB files and Log files


Replication

Master Replica #1

Replica #2…

App App

readwrite read

App

read


In-Memory Configuration

• Trade off between “Performance” and “Durability”

• Memory only configuration

• DB files

• Log files

• Environment files


Performance Example

• Linux : Suse Linux 9.1, AMD Athlon 64 3200+ at 1GHz, 1GB RAM

• Solaris : Sun Solaris 10(x86), SunFire v20z with Dual AMD Opteron at 2GHz, 16GB RAM

• Windows : XP Professional SP2, AMD Athlon 2600+ 2.0GHz, 768MB RAM

Oper/Sec

1,000,000

800,000

600,000

400,000

200,000

1,200,000

Linux Solaris Windows

1 Record Read (DS)

1 Record Write (DS)

1 Record Read (TDS)

1 Record Write (TDS)

1 Record Write (TDS, Write, No-Sync)


Berkeley DB Product Family

1. The original transactional storage engine for applications developed with C, C++, Java, PHP, Perl, etc.

3. Transactional storage engine written in Java and optimized for Java environments

2. Berkeley DB XML provides XQuery-based access to XML documents


Berkeley DB Java Edition

• New product, not a port of the C product

• Same philosophy

• New language for modern runtime

• 100% pure Java

• Single JAR file for ease of distribution, installation

• Runs in same JVM as the application code

• Additional APIs based on Java Collections standard for

transactional collections

• Single process, multi-threaded data access

• Footprint as low as 435KB


Berkeley DB XML

• Built on Berkeley DB, inherits all those features

• Embeddable XML support via XPATH 2.0, XQuery 1.0

• Combine XML and non-XML data in the same database/transaction

• Flexible indexing, cost-based query optimization

• But no full-text search in the current product

• Flexible storage control

• Node

• Whole document

• Optimize for update or retrieval

• Schema support, including validation

• Document streaming from any URI

• DOM-like navigation of documents and result

29

SQL Support

30

SQL SupportHigh Level Overview

• Oracle Berkeley DB (BDB)• The industry-leading open source, embeddable, key-value

database that provides developers with fast, reliable, local persistence with zero administration

• SQLite• A transactional embedded relational database management

system contained in a small C programming library

• Berkeley DB 11gR2 provides the best of both: a small footprint, embeddable, fast, reliable, scalable SQL library with zero administration.

http://www.sqlite.org/index.html

31

SQL SupportA peek inside the box

• Includes sqlite3, JDBC and ODBC

• Implements SQLite 3.6.21 interface - “drop in" replacement for SQLite

• Better concurrency, performance & scalability

• Scalable to terabytes of data, billions of records,

• Berkeley DB reliability - proven in millions of deployments

Yo

ur

Ap

pli

cati

on

Storage Adapter

sqlite3(), ODBC,

JDBC APIs

Tokenizer Parser Generator

Oracle Database

Virtual Machine

Oracle Berkeley DB

SQLite Library

Concurrent B-TreePage Cache Transaction Logs

Oracle Database Mobile Servers

BDB Library

32

SQL Layer Performance


References


Berkeley DB in Open Source

Berkeley DB

All versions

of Linux

All versions

of BSD UNIX

Apache Web Server,

Directory, et al

LDAP Directory

Productivity suite

Chandler email/PIM

Website traffic

analysis

Kerberos Network

Authentication

Database

Subversion version

control system

Red Hat Package

Manager

Internet search

Text editorMail server (MTA)

Mail server (MTA)

SquidGuard

spam blocker

Spam blocker

Perl

Python

GNU C library

PHP

EmailApplicationsInfrastructure Programming

LanguagesToolsOperating

Systems


Berkeley DB Customers

Financial

Services

Service

Providers

Content

Mgmt

Storage &

Sys Mgmt

Enterprise

Infra/EAISecurityNetworking

Telecom

Applications

infrastructure

Devices/

Appliances

Berkeley DB

http://www.emc.com/


Wibro ACR


• 8 Channel Digital Video Recorder

• Features

• 8 Channel video & audio realtime recording capability

• High picture quality : MPEG4

• Remote monitoring and control through network

• Convenient search function with JOG/Shuttle

• Recording during replay

Samsung DVR “SHR 4080”


Motorola A760 A768 A780 E680 Smart Phones

• Requirement: Motorola‟s Linux-based smart phones

needed a data management component to be fast,

flexible, reliable and designed to operate within highly

constrained environments.

• Solution: Motorola selected best-of-breed open

source components including Sleepycat‟s Berkeley DB,

Montavista Linux and Trolltech Qt. Over 3 million of

these devices have been shipped.Motorola A760

Motorola A768

Motorola A780

Motorola E680


Motorola Wireless Network Gateway

Runs on Berkeley DB

• Requirement: Motorola‟s Wireless Network Gateway (WNG) needed carrier-class performance, scalability and reliability. In order to meet 99.999% uptime requirements, the data manager needed to have the ability to automatically restart after a failure with very fast startup and shutdown speed.

• Solution: Motorola selected Berkeley DB to store configuration data such as radio ID and user information for the WNG. Berkeley DB met Motorola‟s requirements for high performance and ability to operate unattended and recover quickly from system crashes.

Ray van Tassle, Senior Staff Engineer, Motorola

“Berkeley DB was 20 times faster than other databases. It has the operational

speed of a main memory database, the startup and shut down speed of a disk-

resident database, and does not have the overhead of a client-server inter-

process communication.”


Cisco Systems Broadband Provisioning

Register Runs on Berkeley DB

• Requirement: Cisco‟s Broadband Provisioning Registrar (BPR) needed to manage up to 5 million networked devices and 150 configuration change transactions/second. BPR needed an embedded data manager which was fast, scalable, reliable and cost-effective.

• Solution: Berkeley DB was used to replace an object-oriented database in BPR. The result was faster, more reliable and saved Cisco $50,000/CPU.

Anton Okmianski, Senior Software Engineer, Cisco Systems

“When we switched to Sleepycat we never looked back. It gave us high

performance, small footprint and a set of well-targeted features that let us

take our solution to a new level.”

Cisco BPR


Openwave Mobile Messaging

Runs on Berkeley DB

• Requirement: Openwave customers demanded a high performance, low TCO solution

• Solution: Openwave uses Berkeley DB for the carrier-class message store and LDAP directory that are the basis for all of its messaging products, including Email Mx, MMSC and IP Voicemail. By eliminating the operator‟s database licensing costs, extra hardware costs and DBA costs, the Berkeley DB based solution provided lower acquisition and operating costs for Openwave‟s customers.

Rich Wong, SVP Products & Solutions Group, Openwave Systems

“Speed, reliability and ease-of-use are important considerations for

telecommunication infrastructure equipment. We’re pleased that Sleepycat

continues to improve Berkeley DB for the telecommunications industry.

Openwave relies on Berkeley DB as the core message store in Email Mx, a

messaging platform that handles more than 1.5 billion messages a day.”


• Requirement: AT&T‟s provisioning software for broadband networks originally used an object oriented database, which was too slow, complex and expensive. Key data manager requirements were:

• Speed to handle the flood of requests following restoration of power after an electrical outage

• Strict transactional integrity to ensure the accuracy of communication with billing and customer care systems

• Solution: Berkeley DB was chosen for use as the backing store for all provisioning operations. Without sacrificing transactional integrity, Berkeley DB provided very fast performance, resulting in significant hardware savings for AT&T‟s customers.

Berkeley DB Powers AT&T Network

Management Systems


• Products Powered by Berkeley DB

• MMSC (Multi-Media Messaging System)

• Highest ranked in Quality of Service

• Highest ranked in Response times

• End-to-end message delivery

• MMSC latency

• WSB (Wireless Service Broker)

• WAP-based services

• Video/Audio streaming

• UOne (Unified communications/messaging)

• Single message store for

• Voice

• Fax

• Email

• MMS

• Video

LogicaCMG Mobile Messaging


EMC Storage

Centera content-aware storage:• Conventional file access• Share duplicate blocks and files to reduce cost

What we store:• Device configuration and logs• File metadata• Hashes and reference counts• ACLs for administration

Why we win:• Scalability• Performance and reliability• Embeddability

berkeley db: data storage for the edge and beyond · 31 sql support a peek inside the box •...

Documents