centera to ecs 2.2.1 native migration (transformation)4 • check details for the specific pools...
TRANSCRIPT
Centera to ECS 2.2.1 Native Migration (Transformation)
VasilyPantyukhin
http://oldhenhut.com
Contents
1. Theoverallprocedure..............................................................................................................2
2. Preparation&Configuration....................................................................................................22.1. Centerapreparation...............................................................................................................................................................22.2. AuthenticationintoECS........................................................................................................................................................72.3. GettheReplicationGroupID..............................................................................................................................................72.4. Createtransformationinstance.........................................................................................................................................8
3. Discovery&Precheck...............................................................................................................93.1. Getprofilemappings..............................................................................................................................................................93.1. Setprofilemapping...............................................................................................................................................................113.2. Settransformationsources...............................................................................................................................................123.3. Runtransformationpre-check........................................................................................................................................123.4. ChecktheECScomponentscreated...............................................................................................................................153.4.1. Namespace.............................................................................................................................................................................153.4.2. Users.........................................................................................................................................................................................163.4.3. Buckets....................................................................................................................................................................................20
3.5. Pre-checkconflicts................................................................................................................................................................213.6. SwitchapplicationtoECS...................................................................................................................................................22
4. RunEnumeration...................................................................................................................22
5. RundataIndexing..................................................................................................................23
6. Rundatamigration................................................................................................................24
7. Runatransformationdatareconciliation...............................................................................25
8. DeletethepreviousTransformationjob.................................................................................27
9. Appendix...............................................................................................................................289.1. CenteratoECSmappings...................................................................................................................................................289.2. Accesscontrolrules..............................................................................................................................................................28
2
1. Theoverallprocedure
TheTransformationandMigrationcontainsseveralphases:
• Preparation&Configuration
• Discovery&Pre-check
• Enumeration
• Indexing
• DataMigration
• Reconciliation
2. Preparation&Configuration
2.1. Centerapreparation
• StartCenteraViewerandthenaccesstheCLIcommands
• GetthelistofCenteraPools
Config#showpoollistPoolNameIDProfilesMask-------------------------------------------------------------------------------…NASforapp_T24abb97ce-1dd2-11b2-97bc-955cedfd9a8d-132rdqeDcw-NASforapp_T34abb97ce-1dd2-11b2-97bc-955cedfd9a8d-142rdqeDcw--------------------------------------------------------------------------------
• CheckthedetailsofCenteraPools
Config#showpooldetailNASforapp_T2CenteraPoolDetailReport------------------------------------------------------Generatedon9June201613:47:19CESTPoolName:NASforapp_T2PoolID:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d-13PoolMask:rdqeDcw-ClusterMask:rdqe-cw--hPoolQuotaAlert:500GBPoolQuotaHardStop:--UsedPoolCapacity:17MBFreePoolCapacity(Alert):500GB
3
FreePoolCapacity(HardStop):--NumberofC-Clips:3NumberofFiles:6Numberofscheduledtasks:0GrantedRightstoAccessProfiles:ProfileNameGrantedEffectiveMonitorCapEnabledHomePool-------------------------------------------------------------------------------NASforapp_T2rdqeDcw-rdqe-cw-noyesyesreplicationrdqeDcw-rdqe-cw-yesyesno-------------------------------------------------------------------------------PoolMappings:nonePoolbeingreplicated:yesConfig#showpooldetailNASforapp_T3CenteraPoolDetailReport------------------------------------------------------Generatedon9June201613:47:24CESTPoolName:NASforapp_T3PoolID:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d-14PoolMask:rdqeDcw-ClusterMask:rdqe-cw--hPoolQuotaAlert:500GBPoolQuotaHardStop:--UsedPoolCapacity:0GBFreePoolCapacity(Alert):500GBFreePoolCapacity(HardStop):--NumberofC-Clips:--NumberofFiles:--Numberofscheduledtasks:0GrantedRightstoAccessProfiles:ProfileNameGrantedEffectiveMonitorCapEnabledHomePool-------------------------------------------------------------------------------NASforapp_T3rdqeDcw-rdqe-cw-noyesyesreplicationrdqeDcw-rdqe-cw-yesyesno-------------------------------------------------------------------------------PoolMappings:nonePoolbeingreplicated:yes
4
• CheckdetailsforthespecificPools
Config#showpooldetailNASforapp_T1CenteraPoolDetailReport------------------------------------------------------Generatedon9June201615:43:21CESTPoolName:NASforapp_T1PoolID:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d-12PoolMask:rdqeDcw-ClusterMask:rdqe-cw--hPoolQuotaAlert:500GBPoolQuotaHardStop:--UsedPoolCapacity:18MBFreePoolCapacity(Alert):500GBFreePoolCapacity(HardStop):--NumberofC-Clips:3NumberofFiles:6Numberofscheduledtasks:0GrantedRightstoAccessProfiles:ProfileNameGrantedEffectiveMonitorCapEnabledHomePool-------------------------------------------------------------------------------NASforapp_T1rdqeDcw-rdqe-cw-noyesyesreplicationrdqeDcw-rdqe-cw-yesyesno-------------------------------------------------------------------------------PoolMappings:nonePoolbeingreplicated:yes • FindthePEAfileforCTApool
• TogeneratenewPEAfileusethecommandupdateprofile<profilename>
<peaversion="1.0.0"><defaultkeyname="NASforapp_T1"><credentialid="csp1.secret"enc="base64">xxxxxx</credential></defaultkey><keytype="cluster"id="4abb97ce-1dd2-11b2-97bc-955cedfd9a8d"name="NASforapp_T1"><credentialid="csp1.secret"enc="base64">xxxxxx</credential></key></pea>
5
• Check.peafilewithJCenteraVerify
• CheckJCenteraVerifylogs
****************************************************************************StartofJCenteraVerifylog********************************************************************************---------------------Generalinformation-----------------------Time: TueJun0716:10:03CEST2016ToolVersion: 3.2.23SDKVersion: 3.2.607---------------------Userconfiguration------------------------TOOLSETTINGSNumberOfFiles: 3FileSize(KB):1,10,100WorkingDir: .\TempRestoreDir: .\Temp\RetrieveLogFile: .\JCenteraVerify.logDeletetempFiles: trueDeleterestoreFiles: true
6
SDKSETTINGSNumberofRetries: -1RetrySleep: -1-----------------JCenteraVerifytestresults-------------------ACCESSNODECONNECTIVITYCHECK192.168.0.6?C:\zz_business_temp\NASforapp_T1.pea isavailable.CLUSTERINFORMATIONClusterId: 4abb97ce-1dd2-11b2-97bc-955cedfd9a8dCentraStar: 4.2.2-4682-2-0AvailableCapacity: 23,240GBTotalFreeCapacity: 26,82GBReplicaAddress: 192.168.0.28CAPABILITIESReadallowed: trueWriteallowed: trueDeleteallowed: trueExistsallowed: truePurgeallowed: falseMonitorallowed: falseQueryallowed: truePrivDeleteallowed: falseWritingthefile11File(s)writtensuccessfully. CA:1V2115AP198S2e01GENR0VQFVS1Readingthefile1. CA:1V2115AP198S2e01GENR0VQFVS11File(s)readsuccessfullyDeletingthefile1. CA:1V2115AP198S2e01GENR0VQFVS11File(s)deletedsuccessfullyWritingthefile22File(s)writtensuccessfully. CA:DDI527S7SB9L5eF51GSNOKM4UIOReadingthefile2. CA:DDI527S7SB9L5eF51GSNOKM4UIO2File(s)readsuccessfullyDeletingthefile2. CA:DDI527S7SB9L5eF51GSNOKM4UIO2File(s)deletedsuccessfullyWritingthefile33File(s)writtensuccessfully. CA:10M3R9KL1OS96eAEJ1LA10VPFVCReadingthefile3. CA:10M3R9KL1OS96eAEJ1LA10VPFVC3File(s)readsuccessfullyDeletingthefile3. CA:10M3R9KL1OS96eAEJ1LA10VPFVC3File(s)deletedsuccessfully*******************************************************************************EndofJCenteraVerifylog******************************************************************************
7
2.2. AuthenticationintoECS
• AuthenticatewithECSkeepingthetokeninthecookiefile
admin@host:/opt/emc>curl-khttps://root:[email protected]/login>/tmp/tokenadmin@host:/opt/emc>cat/tmp/tokenBxxxxxxxxxxxxxxxx0A8=
• Tomakefurtherthecurlcommandsshorter,specifysomeHTTPheadersinthebashvaliable
admin@host:/tmp>exportHEADERS="-HX-SDS-AUTH-TOKEN:$(cat/tmp/token)-HAccept:application/xml-HContent-Type:application/json"
2.3. GettheReplicationGroupID
• CheckallavailableRGsinECSGUI
• WearegoingtousetheRGdistributedacross2xVDCs
8
• GettheIDfortheRG
admin@host:/tmp>curl-s-k-XGET${HEADERS}https://localhost:4443/vdc/data-service/vpools|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><data_service_vpools>…<data_service_vpool><creation_time>1465560162090</creation_time><id>urn:storageos:ReplicationGroupInfo:f3fdeb40-0b14-4896-bdda-2af1710227ec:global</id><inactive>false</inactive><isAllowAllNamespaces>true</isAllowAllNamespaces><description/><enable_rebalancing>true</enable_rebalancing><isFullRep>false</isFullRep><name>rg_ri_st</name><varrayMappings><name>urn:storageos:VirtualDataCenterData:3acbaff8-a2b1-4f0c-b281-27844140e69e</name><value>urn:storageos:VirtualArray:53dd93b6-54e8-4827-bafb-b493d637299d</value></varrayMappings><varrayMappings><name>urn:storageos:VirtualDataCenterData:79c6761c-eadd-4fa8-a05e-69998a49d888</name><value>urn:storageos:VirtualArray:c365c750-b45a-4fcf-993a-a36a6491d941</value></varrayMappings></data_service_vpool></data_service_vpools>
2.4. Createtransformationinstance
• Createtheconfigfile
admin@host:/tmp>vitransformation.json{"type":"centera","name":"CenteraTransformation","description":"Centeratransformation","replication_group":"urn:storageos:ReplicationGroupInfo:f3fdeb40-0b14-4896-bdda-2af1710227ec:global","admin":"admin","password":"centera","management_ip":"192.168.0.6","port":3682,"access_ip":"192.168.0.6","datagram_port":3218}
9
• CreatethenewTransformationjob
admin@host:/tmp>curl–s-k-XPOST${HEADERS}[email protected]://localhost:4443/object/transformation|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
• CheckthenewTransformationjob
admin@host:/tmp>curl-s-k-XGET${HEADERS}https://localhost:4443/object/transformation|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformationList><transformations><id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</id><namespace>centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</namespace><replication_group>urn:storageos:ReplicationGroupInfo:f3fdeb40-0b14-4896-bdda-2af1710227ec:global</replication_group></transformations></transformationList>
• Tomakethecurlcommandsshorter,specifythecommonpartoftheURLintheshellvariable
admin@host:/tmp>exportTRANSFORMATION_BASE_URI=https://localhost:4443/object/transformation/urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d
3. Discovery&Precheck
3.1. Getprofilemappings
AProfileMappingdefinesamappingbetweenapairofdatainstancesdefinedasaCenteraPooland
ProfilemappedtoatoapairofdatainstancesdefinedasanECSbucketandanECSuser.
Auto-calculatedProfileMappingsarelessauthoritativethanmanuallycreatedones.Ifatarget_useror
target_bucketofanauto-calculatedProfileMappingmatchesanalreadyexistingECSuserorECSbucket,
ECSmarksthatauto-calculatedProfileMappingasaconflictingone.Youcannotstartatransformation
ifconflictingProfileMappingsexist.Youcanover-rideaconflictingauto-calculatedProfileMappingby
settingamanualProfileMappingentryforCenterasourceprofile.
10
• ListofcurrentlyavailableProfileMappingentries.
admin@host:/tmp>curl-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}/profile/mapping|xmllint--format–<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><profile_mappings>…<mappings><centera_profile><mask>rdqe-cw-</mask><pool>NASforapp_T2</pool><profile>replication</profile><source_id>NASforapp_T2/replication</source_id></centera_profile><profile_mapping><source_id>NASforapp_T2/replication</source_id><target_bucket>NASforapp_T2</target_bucket><target_namespace>centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</target_namespace><target_user>replication</target_user></profile_mapping><conflict><conflict_type>userExists</conflict_type></conflict></mappings><mappings><centera_profile><mask>rdqe-cw-</mask><pool>NASforapp_T2</pool><profile>NASforapp_T2</profile><source_id>NASforapp_T2/NASforapp_T2</source_id></centera_profile><profile_mapping><source_id>NASforapp_T2/NASforapp_T2</source_id><target_bucket>NASforapp_T2</target_bucket><target_namespace>centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</target_namespace><target_user>nasforapp_t2</target_user></profile_mapping></mappings></profile_mappings>
11
3.1. Setprofilemapping
Manualnamingassignmentandpre-checkconflictsmustberesolvedmanuallybysettingProfile
Mappings.
• SpecifyhowsourceCenteraPool:ProfileshouldbemappedtoECSuserandbacketinthe.jsonfile.
admin@host:/tmp>vimapping-modif.json{"mappings":[
{"source_id":"NASforapp_T2/replication","target_user":"replication3","target_bucket":"NASforapp_T2","target_namespace":"centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d"},{"source_id":"NASforapp_T3/replication","target_user":"replication4","target_bucket":"NASforapp_T3","target_namespace":"centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d"}
]}
• ChangetheAccept:applicationheaderfromxmltojson
• Assignthecustomizedmapping
admin@host:/tmp>exportHEADERS="-HX-SDS-AUTH-TOKEN:$(cat/tmp/token)-HAccept:application/json-HContent-Type:application/json"admin@host:/tmp>curl-s-k-XPOST${HEADERS}[email protected]${TRANSFORMATION_BASE_URI}/profile/mapping
• ForbetterxmlstructurereadabilitychangetheAccept:applicationheaderfromjsonbacktoxml
• Checkethemodifiedmapping
admin@host:/tmp>exportHEADERS="-HX-SDS-AUTH-TOKEN:$(cat/tmp/token)-HAccept:application/xml-HContent-Type:application/json"admin@host:/tmp>curl-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}/profile/mapping|xmllint--format–…<profile_mapping><source_id>NASforapp_T2/replication</source_id><target_bucket>NASforapp_T2</target_bucket><target_namespace>centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</target_namespace><target_user>replication3</target_user></profile_mapping></mappings><profile_mapping><source_id>NASforapp_T3/replication</source_id><target_bucket>NASforapp_T3</target_bucket><target_namespace>centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</target_namespace><target_user>replication4</target_user></profile_mapping>
12
3.2. Settransformationsources
ECSneedtohaveoneorseveralCASpoolstobeselectedforTransformation.Selectionismadeby
supplyingalistofCenteraprofilesource_idvaluesviaSetTransformationSourcesRESTAPI.Onlyone
Centeraprofileperpoolcanbeselectedasasource.SelectedsourceprofileshouldhaveReadandQuery
capabilities
• SpecifytheCenteraPooltobetransformedinthe.jsonfile
admin@host:/tmp>vitransformationsources.json{"source_ids":[
"NASforapp_T2/NASforapp_T2","NASforapp_T3/NASforapp_T3"
]}
• Assigntransformationsources
admin@host:/tmp>curl-s-k-XPOST${HEADERS}[email protected]${TRANSFORMATION_BASE_URI}/transformationSources|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
• Checktheassignedtransformationsources
admin@host:/tmp>curl-s-k${HEADERS}${TRANSFORMATION_BASE_URI}|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><namespace>centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</namespace><replication_group>urn:storageos:ReplicationGroupInfo:f3fdeb40-0b14-4896-bdda-2af1710227ec:global</replication_group><source_ids>NASforapp_T2/NASforapp_T2</source_ids><source_ids>NASforapp_T3/NASforapp_T3</source_ids><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
3.3. Runtransformationpre-check
Duringthepre-checkphase,ECSwillmakesurethedatafromthesepoolscanbemigratedandwill
createthedifferentcomponents(namespace,buckets,users).
Theresultofthepre-checkmustbereadcarefullytoensurethemigrationcanbeperformed
13
Listofthechecksexecuted:
Check DescriptionSOURCE_ADMIN_HAS_CAPABILITIES_CHECK ProvisionedCenteraMAPIuserhasenoughprivilegesDATA_INTEGRITY_CHECK CenterareportsdataintegrityincompletenessHW_VERSION_CHECK CenterahardwareversiondoesnotmetrequirementsUPGRADE_STATUS_CHECK CenteraupgradeisinprogressSW_VERSION_CHECK CenterasoftwareversiondoesnotmetrequirementsCOMPLIANCE_MODE_CHECK UnsupportedcompliancemodePROFILE_MAPPING_CONFLICTS_RESOLVED_CHECK TherearestillunresolvedProfileMappingconflicts
SOURCE_PROFILES_ARE_CONFIGURED NosourcepoolsareaddedtotheTransformationSOURCE_PROFILES_AUTH_CHECK ConfiguredsourcepoolsisnotaccessiblebyprofilegivenNAMESPACE_CREATED_CHECK EnsuresthattargetnamespaceforTransformationbucketsexists,
namespacewillbecreatedifmissing.Newnamespaceadminuserwillbecreatedautomaticallyfornewlycreatednamespace
BUCKETS_CREATED_CHECK Ensuresthattargetbucketsexist,bucketswillbecreatedifmissing
USERS_CREATED_CHECK EnsuresthatECSusersexist,userswillbecreatedifmissingRETENTION_CLASSES_IMPORT_CHECK EnsuresthatECSCASretentionclassesmatchCentera,ECSCAS
retentionpolicieswillbecreatedifmissingCOMPLIANCE_MODE_PERSISTED_CHECK EnsuresthatconfiguredCenteracompliancemodeisstoredin
TransformationmetadataRECONCILIATION_USERS_CREATED_CHECK Ensuresthatmiscellaneousdatauseraccountsneededat
reconciliationphasearecreated
• StartthePrecheckjob
admin@host:/tmp>curl-s-k-XPOST${HEADERS}${TRANSFORMATION_BASE_URI}/precheck|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
• Checkthestatusofthepre-checks
admin@host:/tmp>curl-s-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}/precheck|xmllint--format–<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><precheck_report><processing_speed>0.0</processing_speed><processing_speed_for_last_day>0.0</processing_speed_for_last_day><processing_speed_for_last_hour>0.0</processing_speed_for_last_hour><progress>100.0</progress><start_time>2016-06-09T10:15:58.531</start_time><status>Succeeded</status><prechecks><message/><name>DATA_INTEGRITY_CHECK</name><status>OK</status></prechecks><prechecks>
14
<message/><name>HW_VERSION_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>UPGRADE_STATUS_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>SW_VERSION_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>COMPLIANCE_MODE_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>ADVANCED_RETENTION_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>PROFILE_MAPPING_CONFLICTS_RESOLVED_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>RECONCILIATION_USERS_SETUP_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>SOURCE_PROFILES_ARE_CONFIGURED</name><status>OK</status></prechecks><prechecks><message/><name>SOURCE_PROFILES_AUTH_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>NAMESPACE_CREATED_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>BUCKETS_CREATED_CHECK</name><status>OK</status></prechecks><prechecks>
15
<message/><name>USERS_CREATED_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>RETENTION_CLASSES_IMPORT_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>COMPLIANCE_MODE_PERSISTED_CHECK</name><status>OK</status></prechecks><prechecks><message/><name>RECONCILIATION_USERS_CREATED_CHECK</name><status>OK</status></prechecks></precheck_report>
3.4. ChecktheECScomponentscreated
AllnecessaryECScomponentshavebeencreatedautomaticallyonECS.
3.4.1. Namespace
• CenteraClusterismappedtoECSNamespace
• Namespacenamingconventioniscentera_<CenteraID>
16
• QuotasandAccessDuringOutagearedisabledfortheNamespace
3.4.2. Users
• Objectusersfordataaccessandreconciliationareautomaticallycreated
17
• Usercentera-<CenteraID>
• CASaccessisnotconfiguredforthisuser
• ObjectuserforeveryPool/Bucket
• Thatkindofusersalwaysconfiguredtogetanobjectaccess
• Passwordandcorresponding.peafilearegenerated
• TransformationMetaInfoattributeisspecified
18
• Separatereconciliationuserisusedtocheckdataconsistencyonthefinalstepoftheofdata
migration
19
• Wearenotgoingtousetheuserwhichcorrespondstothe2ndCenteraProfileforthedataaccess
• Despiteonthat,thecorrespondinguseriscreatedfollowingthemappingrules
20
3.4.3. Buckets
• Bucketsare1:1createdcorrespondingtotheCenteraPools
• BucketownerisrootcauseanauthorizationisconfiguredviaACL
• SoftQuotaisassignedaccordingtotheCentera’sPoolQuotaAlertparameter
21
• OfcourseCASfunctionalityisenabled
• MetadataSearchisenablesaswell
• ThemainobjectusewhichcorrespondstothesourceCenteraProfilehasFullControlACLrights
• Additionaluserwhichcorrespondsthe2ndCenteraprofilehasRead/WriteandDeleterights
• Reconciliationusercanreaddataonly
3.5. Pre-checkconflicts
ThereareseveraloutcomesforPrechecks&Provision:
• Veto–ECShasdiscoveredsevereissuewhichpreventsTransformationtoproceed.Issuesmustbe
resolvedandprecheckstoberetriedbeforetheApplicationswitchovertoECS
• Error–ECShasdiscoveredmajorissuewhichmightimpactsomepartofTransformation.Itisunsafe
toperformtheApplicationswitchovertoECSwhileissueexistsasitmaycauseserviceinterruption.
22
Issuesmustberesolvedandprecheckstoberetried,however,incertainscenariosErrorstatuscanbe
ignored
• Warning–Transformationisstarted.ItissafetoperformtheApplicationswitchovertoECS.ECShas
discoveredaminorproblemwhichcanberesolvedafterwards
• Ok-Transformationisstartedwithoutdiscoveringanyproblem
Possiblepre-checkconflictsare:
Check DescriptionuserExists automaticallycalculatedECSusernamematchesalreadyexistingECSuserbucketExists automaticallycalculatedECSbucketnamematchesalreadyexistingECSbucketuserOverlaps severalautomaticallycalculatedECSusernamesequalstoeachother
• Exampleofthepossiblepre-checkconflict
<message>UnresolvedProfileMappingsfound:[NASforapp_T2/replication,...]</message><name>PROFILE_MAPPING_CONFLICTS_RESOLVED_CHECK</name><status>VETO</status>
Themostcommonreasonforpre-checkconflictsisthemappingconflicts.Iscanberesolveasdescribed
inthecorrespondingsectionabove.
3.6. SwitchapplicationtoECS
NowwearereadytoswitchourapplicationtoECS.Dothisnow.
Note:ItestedthatwithCTA.Itworksjustfine.
4. RunEnumeration
Important:Don’tstarttheenumerationbeforetheapplicationhasbeenconfiguredtocommunicatewith
ECSinsteadoftheCentera.Otherwise,newobjectscreatedafterenumerationisstartedwillnotbe
migrated.
AttheEnumerationstageECScreatesalistofCenteracontent
o CenteraQueryisusedtolistclips
o ECSuseswritetimeofthefirstwrittenclipasalowerbound
o Currenttimeasupperboundforthequery
o ECSchopsinitiallydetectedtimeintervalintosmallerpiecestoparallelizeprocessing
23
o TaskstoenumerateresultingtimeframesarespreadacrosstheECSnodesrandomlyto
parallelizeprocessing
o Enumerationofdifferenttimeframesisperformedwithoutstrictordering
o KnowledgeonhowdenselyclipsarelocatedonatimelineisnotavailableforECS–some
enumerationtaskmaygetmoreworkthanothers
• StartEnumerationjob
admin@host:/tmp>curl-s-k-XPOST${HEADERS}${TRANSFORMATION_BASE_URI}/enumeration|xmllint--format–<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
• Checkthestatusofthepre-checks
admin@host:/tmp>curl-s-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}/enumeration|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><enumeration_report><pool_resultsource_id="NASforapp_T2/NASforapp_T2"><entry_count>5</entry_count></pool_result><processing_speed>4.9E-324</processing_speed><processing_speed_for_last_day>4.9E-324</processing_speed_for_last_day><processing_speed_for_last_hour>0.0</processing_speed_for_last_hour><progress>100.0</progress><start_time>2016-06-09T11:53:24.380</start_time><status>Succeeded</status></enumeration_report>
• Callthisrequest,ideallyinatimedloop,untiltheAPIreturnsboth"status":"Succeeded"and
"progress":100.0.
5. RundataIndexing
DuringtheDataIndexingphase,ECSwillgetallthemetadatafromCentera
o AtthisstageECSprocessesthepreviouslycreatedlistofclips/blobsandbuildsinternalindexes
ofCenteracontent
o OnceCenteraobjectisindexeditbecomesnativeECSobject
o IndexedobjectsbecomeavailablegloballythroughallECSVDCsinreplicationgroup
24
o IndexingalsopreparesforupcomingMigrationplanninghowobjectswillbepackedinchunks
• RunIndexingjob
admin@host:/tmp>curl-s-k-XPOST${HEADERS}${TRANSFORMATION_BASE_URI}/indexing|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
• GetIndexingstatus
admin@host:/tmp>curl-s-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}/indexing|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><indexing_report><processing_speed>4.9E-324</processing_speed><processing_speed_for_last_day>4.9E-324</processing_speed_for_last_day><processing_speed_for_last_hour>0.0</processing_speed_for_last_hour><progress>100.0</progress><start_time>2016-06-09T13:08:52.717</start_time><status>Succeeded</status><processed_bytes>18172960</processed_bytes><processed_objects>5</processed_objects></indexing_report>
• Callthisrequest,ideallyinatimedloop,untilitreturnsboth"status":"Succeeded"and"progress":
100.0.
Note:QuotaischeckedduringIndexingforthequotaenabledNamespace/Bucket
o MaycauseIndexingtofailifhardquotaforNamespace/Bucketisexceeded
o Nostoragespacepreallocatedforobjectscontentduringindexing
Incaseanyissueisdetected,thelistofC-clipsthathaven’tbeenmigratedsuccessfullycanbeobtained.
Anymismatchedclipswillnotautomaticallyretry,butrequiremanualinterventiontoretrymigration.
6. Rundatamigration
o Migrationphasegoesthroughallchunkscreatedatindexingphaseandfulfillsitwithobjects
contentMigrationprocessincludeserasurecodingofmigratedcontentonthefly
o MigrationispreformedbyallECSnodes
o Severalchunksaremigratedatthesametimewithoutstrictordering
25
o Duetoconcurrentandorderlessnatureofmigrationsomeobjectsmaybepartiallymigratedat
particularpointintime
Note:YoumustalreadyhavesucceededattheDataIndexingphasetobegintheDataMigrationphase.
• RunMigrationjob
admin@host:/tmp>curl-s-k-XPOST${HEADERS}${TRANSFORMATION_BASE_URI}/migration|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
• GetMigrationstatus
admin@host:/tmp>curl-s-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}/migration|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><migration_report><processing_speed>0.0</processing_speed><processing_speed_for_last_day>0.0</processing_speed_for_last_day><processing_speed_for_last_hour>0.0</processing_speed_for_last_hour><progress>100.0</progress><start_time>2016-06-10T11:21:18.775</start_time><status>Succeeded</status></migration_report>
• Callthisrequest,ideallyinatimedloop,untilitreturnsboth"status":"Succeeded"and"progress":
100.0.
7. Runatransformationdatareconciliation
o DuringReconciliationECSgoesthroughthelistofobjectscollectedduringEnumerationphase
andverifiesthatthosearereadablefromECS
o AstheresultReconciliationphaseproducesalistofobjectswhicharenotreadablefromECSor
theirchecksumdoesnotmatch
o ReconciliationusesspecialECSdatauserswhichhasnomappingtoCenteraprofileswhich
ensuresthatECSlogicwon’tfallbacktoCenteraduringreadingobjects
26
• RunReconciliationjob
admin@host:/tmp>curl-s-k-XPOST${HEADERS}${TRANSFORMATION_BASE_URI}/reconciliation|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
• ChecktheReconciliationstatus
admin@host:/tmp>curl-s-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}/reconciliation|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><reconciliation_report><processing_speed>4.9E-324</processing_speed><processing_speed_for_last_day>4.9E-324</processing_speed_for_last_day><processing_speed_for_last_hour>0.0</processing_speed_for_last_hour><progress>100.0</progress><start_time>2016-06-10T11:39:29.984</start_time><status>Succeeded</status><reconciled_objects>5</reconciled_objects><unread_objects>0</unread_objects></reconciliation_report>
• Callthisrequest,ideallyinatimedloop,untilitreturnsboth"status":"Succeeded"and"progress":
100.0.
• Checkifanymismatchesexist
admin@host:/tmp>curl-s-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}/reconciliation/mismatches|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><reconciliation_mismatches/>
o Re-MigrationphaseisneededtorepeatMigrationforobjectswhichwereidentifiedas
unsuccessfullymigrated
o SinceECSoperatesbyerasurecodedchunksduringmigrationwholechunkre-migrationis
initiatedeventhoughjustasingleobjectcouldbenotmigrated
o ReconciliationcanbeexecutedoncemoretimeassoonasRe-Migrationisfinished
27
8. DeletethepreviousTransformationjob
• CheckthestatusofthepreviousTransformationjob
admin@host:/tmp>curl-s-k-XGET${HEADERS}${TRANSFORMATION_BASE_URI}|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation><namespace>centera_4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</namespace><phases><phase_id>Enumeration</phase_id><status>Succeeded</status></phases><phases><phase_id>Indexing</phase_id><status>Succeeded</status></phases><phases><phase_id>Migration</phase_id><status>Succeeded</status></phases><phases><phase_id>PreCheck</phase_id><status>Succeeded</status></phases><phases><phase_id>Reconciliation</phase_id><status>Succeeded</status></phases><replication_group>urn:storageos:ReplicationGroupInfo:f3fdeb40-0b14-4896-bdda-2af1710227ec:global</replication_group><source_ids>FMA_Site1_T/FMA_Site1_T</source_ids><transformation_id>urn:Transformation:Centera:4abb97ce-1dd2-11b2-97bc-955cedfd9a8d</transformation_id></transformation>
• DeletetheTransformationjob
admin@host:/tmp>curl-s-k-XDELETE${HEADERS}${TRANSFORMATION_BASE_URI}|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformation_phase><phase_id>Deletion</phase_id><status>Pending</status></transformation_phase>
• CheckiftheTransformationjobisdeleted
admin@host:/tmp>curl-s-k-XGET${HEADERS}https://localhost:4443/object/transformation|xmllint--format-<?xmlversion="1.0"encoding="UTF-8"standalone="yes"?><transformationList/>
28
9. Appendix
9.1. CenteratoECSmappings
• CenteraandECSlogicalconfigurationmapping
9.2. Accesscontrolrules
• Accesscontrolmapping
ECSpermission Centeramask CenteracapabilitiesRead r-qe---- Read,Query,ExistWrite ------wh Write,LitigationholdFULL_CONTROL rdqe-cwh Read,Delete,Query,Exist,Clipcopy,Write,LitigationholdPRIVILEGED_WRITE ----D--- PrivilegeddeleteDELETE -d------ Delete
• CenteraAccessMasksarecollapsedintoEffectiveAccessMaskbyfollowingformula
• EffectiveAccessMaskmappedtoappropriateECSBucketAccessrightsforparticularuser
• ECSBucketAccessrightsareautomaticallycopiedintoObjectAccessRightsforeachobject
transformed