effizientes speichern von cold-datakonferenz-nz.dlr.de/pages/storage2016/present/1... · title: how...
TRANSCRIPT
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Effizientes Speichern von Cold-Data
Dr. Dirk Gebh Storage Sales Consultant Oracle Deutschland
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Cold-Data
OHSM Introduction
Use Case – Removing Cold Data from Primary Storage
OHSM and Cloud
OHSM new Features
1
2
3
3
4
5
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Cold-Data
OHSM Introduction
Use Case – Removing Cold Data from Primary Storage
OHSM and Cloud
OHSM new Features
1
2
3
4
4
5
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 5
Infrequent access but significant efford
Cold-Data
Svalbard Global Seed Vault
Example:
=> Log Term Preservation !
„forgotten“ vegetables
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
80% of Data Never Accessed After 90 Days
6
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 7
It depends...
When Is Long Term Preservation Needed?
1. Delete „trash data“ as soon as possible! Don‘t store, don‘t backup, don‘t archive That is most efficient for „trash data“ Deduplication will not solve this problem !
2. Select „worth data“ – It can be hot or cold Put Right Data on Right Storage at Right Time
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 8
Example: Aircraft construction data
Long Time Preserving Data up to 80 Years required
Which storage medium and architecture will meet this requirement?
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Seagate Constellation ES.2 StorageTek T10000D
9
Bit Error Rate of Tape is 10,000 Times Lower Than Disk
How long is data transferred before an uncorrectable error?
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Seagate Constellation ES.2 StorageTek T10000D
10
Bit Error Rate of Tape is 10,000 Times Lower Than Disk
9 days Equivalent to 40 RAID rebuilds per year per disk!
Higher disk capacities = Longer rebuild times = Increasing data loss exposure
165 years
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
“You can have all the tapes in the world, but that won't do you any good without hardware to read them.” – Jory Prum, LucasArts veteran
11
http://www.polygon.com/2015/1/27/7921837/grim-fandango-remastered-interview-double-fine-disney-lucasfilm-sony
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Don’t forget the Cables, Interface and Drivers
12
Old Device Cable Interface Driver Computer
Do Not Ignore It … Do Validate it! Do Migrate it!
Keep up with Technology Changes
?
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
All Data must be Accessable during Life Cycle!
13
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 14
Long Term Preservation
Make sure data is unchanged
Make sure data is valid
?
Checksum
protection
Re-archive
if needed
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Cold-Data
OHSM Introduction
Use Case – Removing Cold Data from Primary Storage
OHSM and Cloud
OHSM new Features
1
2
3
15
4
5
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 16
Oracle Hierarchical Storage Management
Right Data
Right Time
Lower Costs, Improve Efficiencies, Ensure Access
Right Storage
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 17
Archiving with Oracle HSM
• End to end data integrity validation (DIV)
• Fixity
• Open formats
• Product integrations
• High Availability
• Remote Archiving
• OKM support
Ensure Access
• Policy based movement
• Limitless scalability
• Horizontal scaling
• Automated media migration
• Backup and Archiving
• Tiered Storage
• Private Storage Clouds
• Object Cloud Storage
• Oracle Archive Cloud
Lower Costs Improve Efficiencies
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Save 77% by Archiving Rarely Used Data
$1,600,000
$363,000
Backup Only Backup and Archive
Archive Software
Archive Storage
Backup Software
Backup Disk
Primary Disk
18
Save 20x on Backup Software – This savings alone pays for entire archive solution!
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Manage Tiered Storage with Oracle HSM
Ap
plic
atio
ns
Op
en
Stac
k Sw
ift
AP
I St
and
ard
PO
SIX
File
Sy
stem
AP
I
Sto
rage
Ab
stra
ctio
n L
aye
r (O
racl
e H
SM)
Primary Storage
Archive Storage
19
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Improve Efficiency with Policy-based Data Movement
20
Hierarchical Storage Management
and Archive
Management
Types of Policies:
• Archiving, staging, releasing
• Number of copies • Preferred media type • Recycling tape volumes • Media migration • DIV • Fixity
Policy Characteristics:
• Name pattern • Owner • Group • Size • Access age
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Application
How Oracle HSM Simplifies Management
Application writes file to StorageTek QFS file system
Archive copies created based on policy
Application requests file and the file is retrieved from Oracle HSM
1
2
3
Disk Cache Archive Devices
21
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Application
Oracle HSM provides transparency for applications
Application only knows that it wrote a file
And brought it back
1
2
Disk Cache Archive Devices
22
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
• >1 billion files per file system
• Users with 7 billion+ files
• Scale to 1EB+
• Multiple users with 40+PBs
• Achieve 10s of GB/s by scaling horizontally
• Ingest at 5+ GB/s
Achieve Massive Scale with Oracle HSM
Volume Capacity Throughput
23
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Increase Data Ingest Rates by 10X
• Ingest >1 PB of data per day
• Use 10 servers to archive data instead of 1
• Perform faster migration of data to new tape drive technology
• Enable the use of commodity servers
Tape I/O Horizontal Scalability
24
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
License Cost Effective Scale Out Architecture
Enterprise Scalability: Horizontal Tape I/O
QFS Clients
Metadata
Server
(MDS)
Primary
Disk
Tape Storage
Tier
Yellow = metadata / control path
Black = disk data path
Red = tape data path
•Metadata controlled by MDS
•MDS and QFS clients read/write
to disk
•I/O for tape read/write comes
from MDS and QFS clients
•No bottleneck – I/O load spread
evenly across all servers
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle HSM Automated Media Migration
26
Leverage higher density formats with the efficiency of automatic migration
Migrates multiple data streams without impacting production
Seamless file system management
Policy-based for automatic, behind-the-scenes migration
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle HSM Automates Data Integrity Validation
27
Set policy-based
DIV audits
Verify integrity on each read,
write, or any time
Dedicate tape drives for
DIV audits
Self-heals Automatically
Supports StorageTek T10000C/
T10000D and LTO 5/6/7
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 28
Oracle HSM Ensures Long Term Access with Fixity Support
Supports industry standard checksums: MD5, SHA-1, SHA-256, SHA-384, SHA-512
Verifies checksum on each read, write, or copy of a file
Validates file
remains unchanged
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Support movement of data between different systems
Provide protection for data recall separate from the archive system
Simplify migration from one storage medium to another
The Need for Open Formats – Ensure Access
29
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
• Primary Use case: File interchange
• Describes layout of data structures stored on tape
• Introduced in 2010
• ISO standard established in 2016
• Primary Use case: Long term storage
• Collects many files into one archive data set
• Standard established in 1988 and 2001
Oracle HSM supports Open Formats
30
Linear Tape File System (LTFS) Tape Archive (TAR)
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Ensure Future Access with LTFS Format Support
FILE
FILE INDEX
LTFS Format
Oracle HSM Archive
FILE
FILE INDEX
LTFS Format
Export Import
LTFS Access
Import LTFS formatted tapes
Export archive data onto LTFS formatted tapes
31
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 32
Oracle HSM Integrates with WebCenter Content to Ensure Access Throughout Data Lifecycle
WebCenter Content
• Ingest • Access • Collaboration • Search • Reuse
Tiered Storage • Cost Effective • Dynamic • Policy Based • Storage
Virtualization
Storage Experts
Storage Abstraction
Content Experts
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Cold-Data
OHSM Introduction
Use Case – Removing Cold Data from Primary Storage
OHSM and Cloud
OHSM new Features
1
2
3
33
4
5
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Data on Primary Storage Large installed NAS infrastructure – Problem: 80% Cold Data!
Slow NDMP Backup (full / incremental)
all data: hot and cold
Cold data Hot data
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Removing Cold Data from existing Primary Storage Use Case
Existing In
frastructu
re
Oracle Partner Offering
Oracle Offering
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Cold-Data
OHSM Introduction
Use Case – Removing Cold Data from Primary Storage
OHSM and Cloud
OHSM new Features
1
2
3
36
4
5
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Ap
plic
atio
ns
Stan
dar
d P
OSI
X F
ile S
yste
m A
PI
Primary Disk
Capacity Disk
Digital Tape
2. After 30 days of inactivity make copies to tape and/or cloud
3. Make additional copies to tape
Without automation everything stays on expensive primary disk…
With automation via policy…
4. After 1 year remove from disk archive
Automation Is Key Enabler Simplifying Administration & Improving Storage Efficiency
Cloud
1. After 5 minutes create a copy to tape for Data Protection
37
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
OpenStack Swift Object Store Ingest & Access
Private Cloud Infrastructure Solution
Tape Disk Flash
Oracle Hierarchical Storage Manager
Swift
Object store with support for OpenStack Swift
RESTful API
Web interface for tape storage
Make low cost tape easily accessible to web
users
Enable object storage
Custom metadata, no namespace limitation,
near infinite scalability
Eliminate backup software
Deliver a cold storage service directly to end
users
38
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Create On-Premise Storage Clouds with Oracle HSM and OpenStack Swift
39
Tape Archive
Primary Disk
Ap
plic
atio
ns
Op
en
Stac
k Sw
ift
RES
T A
PI
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Cloud Object Storage – Storing on Disk Storage Nodes !
• OpenStack Swift architecture
• Proxy: authenticates Users and routes requests to the appropriate storage nodes
• Account: tracks the containers associated with an account
• Container: tracks the objects associated with a container
• Object: stores objects in the file system
40
3 copies to different storage nodes
GET / PUT / DELETE
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 41
Cloud Object Storage with OHSM Backend
single data copy
up to 4 data copies handeled by OHSM system
External Swift Clients
GET / PUT / DELETE
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Putting It All Together
Cloud Tier
QFS Clients
Metadata Server
On Premise Primary Disk
On Premise Tape Storage
(control path)
Data path
NFS Gateway
Remote Site / Private Cloud
Store data on the optimal storage device in the optimal location
Applications / Data Source
Archive
Application Interfaces (NFS, CIFS, HTTP, FTP, WebDav, OpenStack Swift)
42
Oracle Cloud Software Appliance
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle HSM Featured at Oracle OpenWorld! Announced: Oracle HSM integrates with the cloud for huge cost savings
43
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle HSM Support for Oracle Storage Cloud - Archive Service – Coming Soon! Breakthrough cost for off-site data protection
Archive
Disk Storage
Archive Storage Amazon Glacier
$0.007- 0.013* $/GB/Mo.
* Region dependent
Google Nearline
$0.011 $/GB/Mo.
$0.001 $/GB/Mo.
$12,000/PB/Yr.
Oracle Storage Cloud Service – Archive
Service
44
Oracle HSM
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Cold-Data
OHSM Introduction
OHSM and Cloud
OHSM new Features
1
2
3
45
4
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Strong Investment in Oracle HSM
Data integrity validation (DIV)
WORM enhancements
Stager enhancements
Horizontal scaling
Cloud
Automated DIV
Linear tape file system (LTFS) import/export
Media migration
Fixity
Expanded LUN size
Extended attribute metadata
5.3 6.0 6.1 - New!
46
Up to 5.2
Archive
Virtualization
Automation
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Media Migration
• Accelerates migration
• Frees CPU cycles and personnel
Fixity Enhancement
• Supports standard checksum algorithms
• Verifies file integrity throughout entire lifecycle
Extended Attribute Metadata
• Improves file system performance
Expanded LUN Size
• Supports hardware advancements
• Increases scalability
New Advancements in Oracle HSM 6.1
47
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Solve Migration Challenges with Oracle HSM Automated Media Migration
48
Migration Challenges
Oracle HSM Automated Media Migration
1. Policy-based migration
2. Eliminates data staging
3. Enables parallel migration
4. Reduces resource burden
5. Fastest path to single format
• Time: months or years to complete
• Cost: hardware, network, and personnel
• Complexity: manage multiple formats
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Improved Media Migration with Oracle HSM
49
Migration Challenges
Oracle HSM Automated Media MigrationTM
1. Memory-Assisted Copy
2. StorageTek Direct Copy
• Time: months or years to complete
• Cost: CPU hungry with little business value
• Complexity: manage multiple formats
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Accelerate Migration with Oracle HSM Automated Media MigrationTM
50 50
• Copy to disk and then to tape • Slow, CPU-intensive migration
Current migration method
• Copy directly from tape to tape • Fast/parallel migrate unlimited streams
T10000A/B/C/D LTO3/4/5/6 TS1130/40/50
T10000D
HSM ctrl
dat
a
With Oracle HSM and StorageTek Direct Copy
With Oracle HSM Memory-Assisted CopyTM
• Copy to memory and then to tape • Fast migrate up to four streams
HSM Disk ctrl
data
ctrl
d
ata
ctrl
d
ata
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Migrate All Data or Repack a Portion of Data
51
• Repack only active source files sequentially to a single cartridge
• Migrate entire contents of multiple cartridges to a single cartridge
Migrate Repack
scsi
/sta
tus
dat
a
Active
Expired
Active
Expired
Active
Expired
Active Active Active
scsi
/sta
tus
dat
a
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Migrate All Data or Repack a Portion of Data Automated Media Migration Feature at Scale
52
• Repack only active source files sequentially to a single cartridge
• Migrate entire contents of multiple cartridges to a single cartridge
Migrate Repack
Active
Expired Active
Expired Active
Expired
Active Active Active
scsi
/sta
tus
dat
a
Active
Expired Active
Expired Active
Expired
Active
Expired Active
Expired Active
Expired
Active Active Active
Active Active Active
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Fixity Enhancement
End-to-end fixity Verify checksum
for each read /write /copy
Verify checksum Ingest file and
checksum
53
Standard checksums: MD5, SHA-1, SHA-256, SHA-384, SHA-512
Solaris library utilized for hardware acceleration
Checksum stored in extended attribute
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 54
Dataflow of Setting and Using Fixity
• Ingest Data
•Write to disk cache
•Run ssum or API
•Calculate the hash and store
•OR calculate and compare with given hash
•Update attributes
•Access file for archive
•Validate
•Access file for stage
•Validate
2 1 4 3 5 6
Archive to tape
•Stage to disk cache
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Improved Performance for Extended Attribute Metadata
55
Metadata Tier
Disk Cache Tier
Archive Tiers
Inodes / POSIX Metadata
Extended Metadata Fast Access Data
Archive Data
Oracle HSM 6.0
Metadata Tier
Disk Cache Tier
Archive Tiers
Inodes / POSIX Metadata
Extended Metadata
Fast Access Data
Archive Data
Oracle HSM 6.1
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Max LUN Size Increased 8x
56
16 TB LUN 4 PB
128 TB LUN 32 PB
Prior to 6.1
Oracle HSM 6.1
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 57
Oracle HSM 6.1 HW / SW Support Updates
Solaris 11.3 LTO7
Tape Drives ACSLS 8.4
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Lower Costs
Improve Efficiencies
Ensure Access
Archiving With Oracle HSM
58
Put Right Data on Right Storage at Right Time
Cold - Data
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 59
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.