Transcript
Page 1: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

1

Oracle ZFS Storage Appliance for Exadata Backup & RecoveryHusnu SensoyGlobal Maksimum Data & Information Technologies

Friday, December 14, 12

Page 2: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Husnu Sensoy

• Chief VLDB Specialist in Global Maksimum Data & Information Technologies

• Oracle ACED in BI domain

• Oracle Magazine DBA of the Year 2009

2

Friday, December 14, 12

Page 3: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

• Three strategic areas we focus on

• Complex Event Processing

• Oracle CEP

• Making 500 different business decisions for 1.2 Millions of events in a second

• Data Mining

• Oracle Data Mining and Oracle R Enterprise Edition

• Churn Prediction Models for Telcos

• Marketing Target Selection Models

• Large scale data analytics (what people say Big Data)

• Ten billion rows in a week

• Exadata

• 120+ TB Exadata migration from UNIX systems.

• Exadata Master Class all over the EMEA region for Exadata customers, Oracle partners, and Oracle at the region.

3

Global Maksimum Data & Information Technologies

Just focus on Data & Information in it...

Friday, December 14, 12

Page 4: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Backup & Recovery Challenges of Exadata

Environments

4

• RMAN still does not provide a mechanism to compress image backups

• No footprint optimized way to store multiple copies of the same data

• RAC node utilization during ternary backup

• Backup replication to remote site

Friday, December 14, 12

Page 5: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

ZFS Storage Appliance

5

10.000 feet...

Hardware

ZFS

Oracle Solaris

Friday, December 14, 12

Page 6: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

ZFS Hybrid Storage Pool

6

A Combination of different skills

Application

ZFS Hybrid Storage Pool

ZIL L2ARC Pool

Write SSDs Read SSDs HDDs

Friday, December 14, 12

Page 7: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

RMAN Incrementally Updated Backup Strategy

7

Byte by byte copy of your Exadata

Friday, December 14, 12

Page 8: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

RMAN Incrementally Updated Backup Strategy

7

Byte by byte copy of your Exadata

Image backup/Backupset Level 0

One Week Ago

run {RESTORE DATABASE FROM TAG WEEKLY_FULL_BCKP;

}

Friday, December 14, 12

Page 9: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

RMAN Incrementally Updated Backup Strategy

7

Byte by byte copy of your Exadata

Image backup/Backupset Level 0 #1 #2 #3 #4

One Week Ago Three Days Before

Now

run {RESTORE DATABASE FROM TAG WEEKLY_FULL_BCKP;

}

run {[SET UNTIL SCN x|SEQUENCE x|TIME ‘x’]RECOVER DATABASE FROM TAG DAILY_INC_BCKP;

}

Friday, December 14, 12

Page 10: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

RMAN Incrementally Updated Backup Strategy

7

Byte by byte copy of your Exadata

Image backup/Backupset Level 0 #1 #2 #3 #4

One Week Ago Three Days Before

Now

Recovery Range

run {RESTORE DATABASE FROM TAG WEEKLY_FULL_BCKP;

}

run {[SET UNTIL SCN x|SEQUENCE x|TIME ‘x’]RECOVER DATABASE FROM TAG DAILY_INC_BCKP;

}

Friday, December 14, 12

Page 11: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Storage Overhead of Image Copies

8

ZFS Storage File System Level Compression

One problem with using incrementally updated backup strategy is that RMAN does not provide a mechanism to compress image copies.

But ZFS provides a file system level compression mechanism

Friday, December 14, 12

Page 12: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Compression

9

ZFS Compression vs RMAN Compression over uncompressed data

0

25

50

75

100

None LZJB GZIP L1 GZIP L4 GZIP L9

ZFS Compression

0

25

50

75

100

None LOW MEDIUM

RMAN Compression

Friday, December 14, 12

Page 13: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Flexibility to Travel in Time

10

ZFS Compression can let you to keep multiple image copies ?

Image #1 #1 #2 #3 #4

One Week Ago

Last Night

Now

One Month Ago Two Weeks Ago

#5 #6 #7 #8 #9 #30…#10 #11 #12 #13 #14 #15

Image #2 #8 #9 #30…#10 #11 #12 #13 #14 #15

Image #3 #30…

Image #4 #30

Friday, December 14, 12

Page 14: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Flexibility to Travel in Time (Con’t)

11

ZFS Compression can let you to keep multiple image copies ?

No matter how much you compress, keeping multiple copies of your database is not a clever idea in terms of utilizing your ZFS Storage Appliance.

ZFS has a solution to that problem also: Deduplication

þ

Assume a database of size 10 TB with a daily of 500 GB. By previous slide I wish to store:•1 Full + 1 incremental = 10.5 TB•1 Full + 7 incremental = 13.5 TB•1 Full + 15 incremental = 17.5 TB•1 Full + 30 incremental = 25 TB•Total of 63.5 TB

This value theoretically can be reduced to •1 Full + 30 incremental = 25 TB using deduplication

Friday, December 14, 12

Page 15: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Replication with ZFS Appliance

12

Painless Data Replication

Exadata

ZFS Storage Appliance

ZFS Storage Appliance DR

Exadata DR TernaryStorage

LAN/WAN

Friday, December 14, 12

Page 16: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Optimizing Ternary Backups

13

Silent Tape Backup by NDMP

Exadata

ZFS Storage Appliance

TernaryStorage Exadata

ZFS Storage Appliance

TernaryStorage

NDMPSnapshot

Friday, December 14, 12

Page 17: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Configuration &

Management Tips

14

Friday, December 14, 12

Page 18: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

ZFS Storage Configuration & Management Best

Practices

15

• ZFS Storage Share Configuration

• Remove Update access time on read attribute.

• Do not use cache devices for neither metadata nor data caching.

• Set Synchronous write bias to Throughput

• Ensure that your ZFS Database record size is 128K

• Design multiple shares differentiated depending on their characteristics

• Cleanup unused snapshot & clones.

• Ensure that you use DNFS client.

• Keep in mind that deduplication & ZFS compression require extra CPU power.

• Use RMAN compression whenever possible unless you have a bottleneck on Exadata RAC nodes.

• Prefer LOW or MEDIUM for performance

• To utilize backup parallelism use SECTION option for BIGFILE tablespace data files

Friday, December 14, 12

Page 19: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

16

Monitoring Performance using Oracle Storage Analytics

Keep your eyes on 3 metrics

Friday, December 14, 12

Page 20: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

17

Monitoring Performance using SQL

Query RMAN Catalog Views

set linesize 5000 column filename format a50set pagesize 64

select bai.inst_id, bai.sid, bai.status, buffer_count, trunc((sysdate - open_time) * 24 * 60,2) elaps, substr(filename, instr(filename, '/',1,3)+1) filename, nvl(effective_bytes_per_second, (bytes / ((sysdate - open_time) * 24 * 3600))) / 1024 / 1024 mb_per_sec, to_char(bytes / 1024 / 1024,'09999.99') mb_sofar, to_char(bytes / 1024 / 1024/10.24/32,'999.99') "%", total_bytes / 1024 / 1024 / 1024 total_gb, io_count from gv$backup_async_io bai where bai.type = 'INPUT' and close_time is null order by "%" desc;

Friday, December 14, 12

Page 21: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

18

Backup Performance

A real value based on previous generation 7410

0

500

1000

1500

2000

Before  Tunining +Infrastructure  Tuning +Oracle  Advanced  Compression -­‐Index  Segments

MB/

s

Friday, December 14, 12

Page 22: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Two Real Backup

Strategies using ZFS

Storage Appliance

19

Friday, December 14, 12

Page 23: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

Friday, December 14, 12

Page 24: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

• A system already running on NFS or another storage with a comparable performance.

Friday, December 14, 12

Page 25: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

• A system already running on NFS or another storage with a comparable performance.

• A single disk copy is sufficient.

Friday, December 14, 12

Page 26: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

• A system already running on NFS or another storage with a comparable performance.

• A single disk copy is sufficient.• Quick recovery from failure is necessary in case of a primary storage loss.

Friday, December 14, 12

Page 27: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

• A system already running on NFS or another storage with a comparable performance.

• A single disk copy is sufficient.• Quick recovery from failure is necessary in case of a primary storage loss.• Database size very small with compared to ZFS Storage pool size.

Friday, December 14, 12

Page 28: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

• A system already running on NFS or another storage with a comparable performance.

• A single disk copy is sufficient.• Quick recovery from failure is necessary in case of a primary storage loss.• Database size very small with compared to ZFS Storage pool size.

Open Storage7000

FRA

Controlfile AutobackupArchivelogsIncremental

BackupsetsImage Copies

Friday, December 14, 12

Page 29: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

• A system already running on NFS or another storage with a comparable performance.

• A single disk copy is sufficient.• Quick recovery from failure is necessary in case of a primary storage loss.• Database size very small with compared to ZFS Storage pool size.

Open Storage7000

FRA

Controlfile AutobackupArchivelogsIncremental

BackupsetsImage Copies

ALTER SYSTEM SET DB_RECOVERY_DEST=‘/export/fra’;

Friday, December 14, 12

Page 30: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

• A system already running on NFS or another storage with a comparable performance.

• A single disk copy is sufficient.• Quick recovery from failure is necessary in case of a primary storage loss.• Database size very small with compared to ZFS Storage pool size.

Open Storage7000

FRA

Controlfile AutobackupArchivelogsIncremental

BackupsetsImage Copies

ALTER SYSTEM SET DB_RECOVERY_DEST=‘/export/fra’;

RUN { RECOVER COPY OF DATABASE WITH TAG ‘DAILY_BACKUP'; BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG ‘DAILY_BACKUP‘ DATABASE;

}

Friday, December 14, 12

Page 31: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Simple Design

20

Creating your FRA on ZFS Storage Appliance

• A system already running on NFS or another storage with a comparable performance.

• A single disk copy is sufficient.• Quick recovery from failure is necessary in case of a primary storage loss.• Database size very small with compared to ZFS Storage pool size.

Open Storage7000

FRA

Controlfile AutobackupArchivelogsIncremental

BackupsetsImage Copies

ALTER SYSTEM SET DB_RECOVERY_DEST=‘/export/fra’;

RUN { RECOVER COPY OF DATABASE WITH TAG ‘DAILY_BACKUP'; BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG ‘DAILY_BACKUP‘ DATABASE;

}

RUN { ALTER DATABASE MOUNT;SWITCH DATABASE TO COPY;RECOVER DATABASE;ALTER DATABASE OPEN;

}

Friday, December 14, 12

Page 32: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

Friday, December 14, 12

Page 33: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option

Friday, December 14, 12

Page 34: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

Friday, December 14, 12

Page 35: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

• To Just now

Friday, December 14, 12

Page 36: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

• To Just now• To somewhere in last week

Friday, December 14, 12

Page 37: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

• To Just now• To somewhere in last week

• ZFS Storage pool is at comparable size with production size.

Friday, December 14, 12

Page 38: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

• To Just now• To somewhere in last week

• ZFS Storage pool is at comparable size with production size.

Open Storage7000

FRA

Image Copies

INC

Incremental Backupsets

ARCH

Archivelogs

LZJB

DEDUP

LZJB

Friday, December 14, 12

Page 39: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

• To Just now• To somewhere in last week

• ZFS Storage pool is at comparable size with production size.

Open Storage7000

FRA

Image Copies

INC

Incremental Backupsets

ARCH

Archivelogs

LZJB

DEDUP

LZJB

ALTER SYSTEM SET DB_RECOVERY_DEST=‘/export/fra’;

Friday, December 14, 12

Page 40: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

• To Just now• To somewhere in last week

• ZFS Storage pool is at comparable size with production size.

Open Storage7000

FRA

Image Copies

INC

Incremental Backupsets

ARCH

Archivelogs

LZJB

DEDUP

LZJB

ALTER SYSTEM SET DB_RECOVERY_DEST=‘/export/fra’;

RUN { RECOVER COPY OF DATABASE WITH TAG ‘DAILY_BACKUP'; BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG ‘DAILY_BACKUP‘ DATABASE TO DESTINATION ‘/export/inc’;

}

Friday, December 14, 12

Page 41: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

• To Just now• To somewhere in last week

• ZFS Storage pool is at comparable size with production size.

Open Storage7000

FRA

Image Copies

INC

Incremental Backupsets

ARCH

Archivelogs

LZJB

DEDUP

LZJB

ALTER SYSTEM SET DB_RECOVERY_DEST=‘/export/fra’;

RUN { RECOVER COPY OF DATABASE WITH TAG ‘DAILY_BACKUP'; BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG ‘DAILY_BACKUP‘ DATABASE TO DESTINATION ‘/export/inc’;

}

RUN { ALTER DATABASE MOUNT;RESTORE DATABASE FROM TAG ‘DAILY_BACKUP’RECOVER DATABASE;ALTER DATABASE OPEN;

}

Friday, December 14, 12

Page 42: Oracle ZFS Storage Appliance for Exadata Backup & · PDF file1 Oracle ZFS Storage Appliance for Exadata Backup & Recovery Husnu Sensoy Global Maksimum Data & Information Technologies

Advanced Design

21

Multiple Image Copies for Multiple Recovery Points

• Quick recovery using SWITCH is not an option• Two recovery capabilities are necessary

• To Just now• To somewhere in last week

• ZFS Storage pool is at comparable size with production size.

Open Storage7000

FRA

Image Copies

INC

Incremental Backupsets

ARCH

Archivelogs

LZJB

DEDUP

LZJB

ALTER SYSTEM SET DB_RECOVERY_DEST=‘/export/fra’;

RUN { RECOVER COPY OF DATABASE WITH TAG ‘DAILY_BACKUP'; BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG ‘DAILY_BACKUP‘ DATABASE TO DESTINATION ‘/export/inc’;

}

RUN { ALTER DATABASE MOUNT;RESTORE DATABASE FROM TAG ‘DAILY_BACKUP’RECOVER DATABASE;ALTER DATABASE OPEN;

}

RUN {SET COMPRESSION ALGORITHM ‘MEDIUM’;RECOVER COPY OF DATABASE WITH TAG ‘WEEKLY_BACKUP‘ UNTIL TIME ‘SYSDATE-7’; BACKUP AS COMPRESSED BACKUPSET INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG ‘WEEKLY_BACKUP‘ DATABASE TO DESTINATION ‘/export/inc’;

}

Friday, December 14, 12


Top Related