Storage Accounting

John Gordon, STFC

GDB March 2013

• History• Timeline• Current state• Evidence of testing• Current issues • Open Issues• Portal Views• Reports?

• AOBData RetentionPortal Requirements

Outline

3

EM

I IN

FSO

-RI-

26

16

11

EMI StAR

• EMI recognised the requirement for storage accounting• EMI created a proposal for a storage accounting record

(StAR)• ‘Finalised’ in June 2011• Revised Autumn 2012 to include the WLCG

requirements I described at the GDB last June.• Published version: http://cds.cern.ch/record/1452920?

ln=en• Been through public comment in OGF and just

resubmitted for approval• implemented by EMI storage providers in EMI 3

Original EMI StAR timeline• May 2011: StAR definition agreement• Oct. 2011: Each SE to provide an implementation plan• May 2012: Each SE to implement accounting sensors• June 2012: Accounting record publishers (APEL and/or

DGAS) to publish storage records• December 2012: Each SE to test and deploy their

accounting sensors (for EMI 3 release)• January-April 2013: Bug-fixing in EMI 3 RC’s• April 2013: Accounting sensors released in EMI 3

4

EM

I IN

FSO

-RI-

26

16

11

Actual EMI Timeline

• February 2013 – first test storage records appeared from dcache and dpm

• February-March 2013 checking and revising.• 11 March – Release of EMI 3.0 • Any changes required should be available in

update 1.

Status of StAR implementations

• Dcache – published from a test system.• Dpm - published from a test system.• StoRM – did not make EMI 3.0 cutoff.• NGI-Italy is working on publishing for other storage

systems from the BDII.• EMI produced tests to verify that StAR records agree

with standardBut that doesn’t address requirements on semantics.

• StAR publishing uses the same SSM infrastructure as the new EMI cpu client, and the current OSG, DGAS, SGAS, CERN, Lyon, CPU publishing

Issues from Testing

• No GOCDB Sitename – may be artifact of test sites.

• No ResourceCapacityAllocated – just Used

• Under discussion

Open Issues

• Frequency of publishingSites could collect data frequently and publish

averages less frequently, say dailyRepository could average daily records into months,

like CPU.• What is to be viewed in the portal?

• Summarise data by • Site, VO, Month (Average, min, max)• Views at any point in tree of

Tier1, Tier2, Country, NGI• View Average Storage for a time period and VO

selection, for any two of (site/country/Tier2), VO, Date

• E.g. Country(date), VO(Site), VO(date), Site(date) (similar to CPU Portal Views.)

• WARNING! Examples do not show real data.

Accounting Portal

Portal View for Storage

• Storage per country per month

• Storage used by LHC VOs at French sites.

Allocated and used.

Storage for Alice in France

• When storage accounting is deployed at most sites then:-The Tier1 monthly report can be autofilled with

storage data.The Tier2 report can be extended to cover storage as

well as cpu.

Reports

• Verify that storage providers are publishing consistent records.

• EMI 3.0 will allow records to be published to extend testing of client, transport, and repository

• Resultant fixes get incorporated in a future EMI3 update (March or April).

• Wider coverage of storage solutions StoRM and BDII

• Finalise what is shown in portal• Implement Portal view and reports.

Next Steps

• Storage Accounting is making (slow) progress. • There will not be much data to show the CRRB

but it can be reported that it is happening.• Are sites and experiments happy with

suggestions for portal displays?• Are there suggestions of other views or

reports?• Questions?

Conclusions

APEL – Data Retention• Last year EGI discussed the APEL Data Retention Policy which stated that

user data should not be kept for longer than 12 months. We agreed to extend this to 18 months.

• We are just getting round to implementing this and want to clarify the implications.

• We keep the anonymous summaries of all records from the beginning of time (2004) and will continue to do so. This is what you can view today in the Accounting Portal

• In future we will delete individual Job Record data older than 18 months at RAL. i.e. We do not keep anonymous job records without the UserDN for longer.

• Highlighting this in case there is any misunderstanding. Today, if you have the privileges, youcan see UserDN data going back more than a year.

• UserDN data older than 18 months will be deleted on 1st July and every month thereafter.

20

... Just one more thing.

• EGI takes requests for features in the Accounting Portal.

• If you have any requests for different views or reports, please discuss them with me, or circulate round GDB mailing list and we can work up the requirement.