storage workshop summary wahid bhimji university of edinburgh on behalf all of the participants…
TRANSCRIPT
![Page 1: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/1.jpg)
Storage Workshop Summary
Wahid Bhimji University Of Edinburgh
On behalf all of the participants…
![Page 2: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/2.jpg)
What was it?
• Monday / Tuesday this week.• Focused on Tier 2 Grid Storage in the UK.• Good attendance from UK storage
community:– All our SRM technologies dCache/DPM/ Storm – Expertise in the demanding customers - CMS/
ATLAS – but not forgetting the others
![Page 3: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/3.jpg)
Stated Aims
• Data at LHC– Tier 2 storage facing significant challenges.
• Workshop - take a step back and: 1.Identify significant issues from site experiences. 2. Provide training and spread good practice for
current SRM/Storage technologies. 3.Consider other/emerging technologies:– Do these actually address the challenges – If so – what do we need to do to test/develop them.
![Page 4: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/4.jpg)
Overview
Sessions on: • Site Experiences – bottlenecks• SRMs – experiences/tutorials/plans/ comparisons• Hardware• Filesystems and File Access• “The Future”
Some highlights here – see upcoming write-up for full info…
![Page 5: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/5.jpg)
Site Experiences
• Sites (big and small) running well• Dealing with all kinds of
operational issuesIssues:• Upgrade schedule – can gridpp
manage process, to getting features and stability
• Thin(ner) SRM layers - do all ‐ use cases need the full SRM spec - longer term discussion‐
![Page 6: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/6.jpg)
SRM – Storage Resource Manager
Variety of SRM implementations (3 in UK Tier2s) • All increasingly mature – no losers dropping out• Do have different strengths
![Page 7: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/7.jpg)
SRMs
dCache:•Getting easier to use •Well liked by current sites•Certain key valuable features such as replication
DPM:•Most used in UK ( and world)•Support from CERN improved• Development of some features desirable (replication, draining , checking) we can push /help them
StoRM •“Thin” SRM - exploits filesystem•Now mature and more widely used•Questions over support and feature implementation •Needs increasing UK expertise
![Page 8: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/8.jpg)
SRMs
• Converging (in some ways)– eg NFS4, file://• But is there a point in making DPM more like
dCache (in features) or Storm (in Lustre support) …
![Page 9: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/9.jpg)
HardwareSSDs: • Ongoing Studies at
Glasgow• Performance for sure
but is it needed if LHC data structures and access improve
Networking: • 10 Gig for those who
can afford it… channel bonding for others
![Page 10: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/10.jpg)
Procurement Discussions
• Advantages of teaming together with “others” (inc university or T2 or T1) for hardware purchases to get bulk rates.
• Even sharing purchase information (where allowed) is useful.
![Page 11: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/11.jpg)
Filesystems – File Access
File access:• Changes in LHC file access
have big impacts on site storage efficiencies, tuning etc.
• Need to continue studies and keep in close contact with developments
![Page 12: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/12.jpg)
Filesystems
• Other advanced filesystems - GPFS, HDFS can offer features that can be exploited for HEP storage
• As can models of analysis such as Hadoop Map-Reduce
![Page 13: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/13.jpg)
“The Future”
• Brainstorming session covered both short term needs and longer term strategy
• Compile to proper output -> workplan but, to give some examples…
![Page 14: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/14.jpg)
Discussions
• Local file access• User space
expectations• Upgrade
schedule getting the features and stability
•Super lightweight SRMs•Current solutions made more easily deployable and configurable
•Support: Recipes •gridpp certified?
•Coordinated testing of implementations •Regular Workshops!
Emerging Issues / QuestionsEmerging Issues / Questions
Dream storage solutions
Dream storage solutions
Actions for GRIPP STORAGE
Actions for GRIPP STORAGE
![Page 15: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/15.jpg)
Conclusions
• Valuable workshop – In and outside sessions
• Indications of challenges ahead –And work to be done
![Page 16: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/16.jpg)
Conclusions
People say “storage is hard”– So we are lucky to have the GridPP Storage
Community– We need to maintain fora for it.
![Page 17: Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…](https://reader036.vdocuments.site/reader036/viewer/2022070305/5515c2f8550346a3758b4730/html5/thumbnails/17.jpg)
Some Issues• Local file access – random file patterns cause load• Priorities for Evaluation – Hardware / SRM-Filesystems• Which SRM is best for a Tier 2- not just a big=dCache small=
DPM issue.• Do I buy SSDs? 10 Gig E? More reliable disks? • User space expectations- “I have 2 TB in my pocket why do
you give me 1 on “the grid”• Should GridPP package/certify its own releases – ensure
they are stable + get early access to features• Do we need (to use) a lighter way to access some things on
a grid – at Tier 3s etc. • Can we get more value through shared procurements