1 © copyright 2007 emc corporation. all rights reserved. centera integration training advanced...

1© Copyright 2007 EMC Corporation. All rights reserved.

Centera Integration Training

Advanced Centera Topics

Centera Corporate Systems Engineering

Sept 2007

Centera Integration Training: Advanced Centera Topics


Agenda

Retention Management– Advanced Retention Management features

Retention Governors Event Based Retention Litigation Hold

– Extending Retention (non-EBR)

Replication of Mutable Metadata

Content Addresses– Standard Formats– ClipID Canonical Form

SDK Logging

Wrap Up and Questions



Advanced Retention Management

Introduced in CentraStar / SDK 3.1

Separate licence

3 main features – Retention Governors– Event Based Retention– Litigation Hold



Enforced Retention

The presence of retention information can now be enforced– reject C-Clips without retention period or class.

Applies to pre-3.1 SDK only 3.1 SDK adds Default Retention Period if Application does not

– disabled by default

Valid retention value constraints can also be added– Min / Max retention Governors set on CentraStar– SDK enforces RetentionPeriod values to fall within the range



Min / Max Retention Governors Applied on Write, Replication and Restore

– Only applies to Retention Period values– C-Clips with a Retention Class are not checked against Min / Max– Not examined during Delete

Available on a per pool basis SDK Returns an error if value is outside the Min/Max

constraints:– FP_RETENTION_OUT_OF_BOUNDS_ERR

FPPool_GetCapability(FP_RETENTION, attribute) can check the limits

– FP_FIXED_RETENTION_MIN– FP_FIXED_RETENTION_MAX– FP_VARIABLE_RETENTION_MIN– FP_VARIABLE_RETENTION_MAX



Advanced Retention Management – Retention Governors

Application

SDK

CentraStar

Application creates C-Clip

SDK checks C-Clip

Retention InformationPresent?

SDK throws error

Server persists CDF and returns CA

Application deletes C-Clip

Retention Fits in Fixed Min/Max?

Apply defaultretention for the pool

(Infinite if CE+)

SDK writes C-Clip

Server deletes C-Clip



Advanced Retention: Default Values

Default values for all pools after enabling license:– Fixed retention minimum 0, maximum infinite– No enforced presence of fixed retention– Event based retention minimum 0, maximum infinite

Default Retention Periods must be within the Min/Max settings if present (except max in CE+, which can exceed it)



Event Based Retention (EBR)

Fixed retention can still be specified as before (period or class) – Must now comply with rules for Min/Max Governor

EBR can also be set on new C-Clips or on copies of existing C-Clips Disabled by default

– Enabled using FPClip_EnableEBRWithPeriod / Class Period must also comply with rules for Min/Max Governor

Stored in the C-Clip as before (immutable) No default value, but period or Class must be specified up-front



Event Based Retention (EBR)

EBR values applied when “triggered” once by a future event– FPClip_TriggerEBREvent uses the policy set when C-Clip created

New policy can be applied instead– FPClip_TriggerEBREventWithPeriod /Class– The longer of the old and new values will be used

New value only applied if longer than original setting

Must comply with rules for Min/Max Governor Event Date is set by SDK based on the cluster time Note: EBR information is not exported / imported by the Raw API

commands



Event Based Retention – Reporting

FPClip_IsEBREnabled:– Return TRUE if EBR is enabled, FALSE otherwise.

FPClip_GetEBREventTime:– Returns the time the EBR event was triggered– 0 if the event has not been triggered or the C-Clip is not EBR

enabled.



Event Based Retention – Reporting

FPClip_GetEBRPeriod:– Returns the greater EBR period associated with the clip

Initial value when C-Clip created / value set on Event trigger

– If EBR set with a class, the value of the class is used– 0 returned if the Clip is not EBR enabled.

FPClip_GetEBRClassName:– Returns name of the EBR retention class associated with the clip– Returns empty string if no class is set– … or if EBR period set on clip > period value of the class



Event Based Retention - Expiry

Deletion is possible when:– Event has occurred– AND Current time > creation date + fixed retention– AND Current time > event date + variable retention

Note: Privileged Delete can still overrides these rules.

C-ClipCreated

Time

Event

fixed retention

fixed retention

variable retention

variable retention

delete allowed

delete allowed

event not specifiedC1C1

event not specifiedC2C2

fixed retention

variable retentionevent not specifiedC3C3

delete allowed



Litigation hold

Litigation Hold(s) can be set / removed at C-Clip level– Retention is held indefinitely until removed.– Privileged Delete cannot override the Hold!

When a C-Clip is put under litigation hold via FPClip_SetRetentionHold the server tracks (outside the CDF):

– Litigation identifier – text string uniquely identifying the Hold– Litigation hold flag – true/false– Date/Time when the C-Clip was put under litigation hold.– Profile that put the C-Clip under litigation hold.



Litigation hold

C-Clip is released from a Hold by setting the hold flag to false

A C-Clip can be under multiple Litigation Holds at any time.

FPClip_GetRetentionHold returns TRUE if there are active Hold(s) on a C-Clip

Note: Cannot query for C-Clips that have active Hold(s) or obtain keys of C-Clips on hold



Advanced Retention Management – Event Based Retention

Application

SDK

CentraStar

Application creates C-Clip

SDK checks C-ClipSDK throws error

Server persists CDF and returns CA

Apply defaultretention for the pool

(0 if CE+)

SDK writes C-Clip

Application sets event

SDK checks event data

Variable Retention

InformationPresent?

Retention InformationPresent?

Retention Fits in Variable

Min/Max?

Retention Fits in Fixed Min/Max?

Server processes event

SDK sends event



Centera has a mechanism for placing a C-clip under fixed retention

Once a C-clip has been placed under fixed retention, the retention period cannot be modified

Customers may want to place a document under short term temporary retention, and then extend the retention period multiple times in the future

Note: this approach works well for extending the retention period– Decreasing retention may require Privileged Delete

To delete older CDF versions where retention has not yet expired

– Not allowed on CE+

Extending Retention without EBR



C-clip is created with a retention period / class– e.g. 30 days

Application stores the content address of the C-Clip– e.g. in local database

Extending Retention – Step 1 – C-Clip is created

blob

DQG22UC39LHJOe66R2K2JIBD4EF

Client Database

CDFretention = 30 days



At some point in time, retention requires to be extended– C-Clip is opened– New retention value is set in the CDF e.g. 60 days– C-Clip is written and the new ContentAddress is stored by the application

Replaces the original content address in the application database Standard SDK metadata in the new C-Clip contains old content address

Extending Retention – Step 2 – Retention is extended

DQG22UC39LHJOe66R2K2JIBD4EFEFG242769LH32e57R23E2IBC4FE

Client Database

blob



prev.clip=DQG22UC39LHJOe66R2K2JIBD4EF



Retention could be extended multiple times– EBR can only extend retention once

Extending Retention – Step 3 – Further extension of Retention

blob

EFG242769LH32e57R23E2IBC4FEE3423ABE3D5EBe34E3D32156A3a

Client Database CDFretention = 60 days

CDFretention = 7 years

prev.clip=EFG242769LH32e57R23E2IBC4FE




C-Clip has had retention extended several times– “List” of CDFs to delete

Traverse the list deleting the CDFs– use previous clip ID as the “pointer” to the “ancestor” CDF

FPClip_GetDescriptionAttribute (clipID, “prev.clip”, outputBuffer, size)

– Original CDF has empty value for previous Clip ID

Blob is now unreferenced and will be Garbage Collected

Extending Retention – Step 4 - Deletion

blobCDF

retention = 60 days

CDFretention = 7 years

prev.clip=EF242769LH32e57R23E2IBC4FE


prev.clip=DQG22UC39LHJOe66R2K2JIBD4EF

prev.clip=



Replication of Mutable Metadata

Replication will replicate any change in EBR and LH information

Profile used for replication needs several capabilities– clip-copy (c) for regular replication– delete (d) for global delete– privileged-delete (D) global privileged delete– clip-copy (c) for EBR information– litigation-hold (h) for LH information



Content Address Formats

Naming Scheme SizeSingle instance

Performance Enhanced?

Enhanced Collision Resistance?

M (MD5) 27 Yes No No

MG (MD5+GUID) 53 No No Yes

GM (w/o Discriminator) 53 No Yes No

GM+D (GM with Discriminator)

53 No Yes Yes

M++ (MD5 + 128b SHA-256) 53 Yes No Yes

SDK Collision Avoidance Disabled SDK Collision Avoidance Enabled

Cluster Storage Strategy Clip Blob Clip Blob

SSC (Capacity) M M (or M++) MG MG

SSPP (Performance Partial) M GM M GM+D

SSPF (Performance Full) GM GM GM+D GM+D

From 2.4 CentraStar, MG and GM forms are no longer used.



ClipID Canonical Format

Binary encoding for Content Address

Enables independence from character encoding specific to, e.g., Mainframe and Open Systems

– Content addresses generated by Mainframe use EBCDIC encoding!

Reduces byte count of CA from 65 bytes to 40 bytes– Handy for saving space, e.g., in a database record– Is not NULL-terminated

FPClipID_GetCanonicalFormat (const FPClipID inClipID, FPCanonicalClipID outClipID)

FPClipID_GetStringFormat (const FPCanonicalClipID inClipID, FPClipID outClipID)



SDK Logging

Highly configurable by category– Debug messages / API calls / Warnings / Errors

… component– POOL, RETRY, XML, API, NET, TRANS, PACKET, EXCEPT– REFS, MOPI, STREAM, CSOD, CSO, MD5, SHA– APP (for application level logging)– LIB (for library loading)

and file properties– Format / Path / Size / Overflow (backups)



Enabling SDK Logging with Configuration File

FPLogState.cfg file in application directory– Default method– Behaviour is dynamically controllable via changes in file state

Enable by adding file to directory Disable by deleting file from directory Alter by editing existing file in directory

– Polling interval is configurable (default is 5 minutes)

FP_LOG_STATE_PATH environment variable– SDK tests for this setting only if default configuration file not found– determines non-default location for configuration file

Behaviour is then the same as above



Enabling SDK Logging via Environment Variables

FP_LOGPATH – location of the output file

FP_LOGKEEP – creation mode– CREATE / APPEND / OVERWRITE

FP_LOGLEVEL – category of messages to log– FP_LOGLEVEL_ERROR / WARN / LOG / DEBUG

FP_LOGFILTER – components to log messages from

FP_LOGFORMAT – XML or TAB formatting

FP_LOG_DISABLE_CALLBACK– Disable the use of any application supplied logging callback method

FP_LOG_MAX_OVERFLOWS – number of backup logs

FP_LOG_MAX_SIZE – maximum size of log file

FP_LOG_STATE_POLL_INTERVAL– Polling frequency in minutes, default=5 (-1: off, 0: on each log event)



Configuring SDK Logging using API Calls

FPLogStateRef object holds current status– Get / Set methods change the property values– Serializable i.e. can be saved to or loaded from file

API calls to change logging status dynamically– FPLogging_Start(FPLogStateRef)– FPLogging_Stop()

Application can also log messages to the SDK logfile– FPLogging_Log(const FPLogLevel, const char *)

… or register its own logging method for the SDK to use– FPLogging_RegisterCallback(FPLogProc)– Can be disabled if required



Questions?

Please Visit SDK Forums on Centera Developer’s Portal– http://lighthouse.emc.com

Corporate Systems Engineering Team provides:– Info, Training, Whitepapers– Design Advice / Design Reviews– Debugging Support– Code Reviews

Email Support Available from:– [email protected]

http://lighthouse.emc.com/

1 © copyright 2007 emc corporation. all rights reserved. centera integration training advanced...

Documents