![Page 1: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/1.jpg)
Lofar Information System on GRID
A.N.Belikov
![Page 2: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/2.jpg)
Lofar Long Term Archive
Prototypes: EGEE Astro-WISE
Requirements to data storage
Tiers
Astro-WISE adaptation
Problem: optimal data placement
![Page 3: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/3.jpg)
Lofar: Science
![Page 4: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/4.jpg)
Lofar: Network
10 Gbps optical
single data processing center for raw data (CIT, Groningen)
![Page 5: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/5.jpg)
Lofar: Data Processing
GbE
sw
itch
GbE
switc
hG
bEsw
itch
GbE
switc
hG
bEsw
itch
GbE
switc
h
Auxiliary processingsection
Stor
age
syst
em
Off-line processing/user software
Off-lineStorageOn-Line
Long term Archiving& Data ExportG
bEsw
itch
On-Line pipeline processingBlue Gene/L
WAN
Blue Gene/L
linux cluster
centralized data processing for raw data
long-term data storage
![Page 6: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/6.jpg)
Lofar: LTA requirements
![Page 7: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/7.jpg)
Lofar: LTA requirements
20 PB for 5 yearsdecentralized data storagedecentralized data reprocessinginternational partners
![Page 8: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/8.jpg)
LHC & EGEE
15 PB data per year grid computing
EGEE: ~250 sites, >45000 CPU
www.eu-egee.org EGEE-III
![Page 9: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/9.jpg)
EGEE: Tiers approach
![Page 10: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/10.jpg)
Astro-Wise: grid
![Page 11: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/11.jpg)
Astro-WISE: dataservers (1)
![Page 12: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/12.jpg)
Astro-WISE: dataservers (2)
Astro-WiseLoWise
dataserverdata retrieved by URI0.5 TB 7 sec
3.5 TB 20 sec 1 PB 8 h
![Page 13: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/13.jpg)
LoWISE: URI storage
![Page 14: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/14.jpg)
LoWISE: Tiers
Astro-WISE-like interfaces user access (from PM to anonymous) Tier 1 as core element
![Page 15: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/15.jpg)
LoWISE: data storage
distributed file systems: GPFS (IBM) Lustre (Sun)
dCAcheCASTOR
![Page 16: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/16.jpg)
Lofar-WISE: Data Storage
initial configuration
distributed FS & EGEE
Oracle for metadata
![Page 17: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/17.jpg)
Problem: users
users: observer, project managerprivileged users: operators
![Page 18: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/18.jpg)
Problem: CE & SE geographical location file transfer user access research group driven
![Page 19: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/19.jpg)
Problem: Access policy
![Page 20: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/20.jpg)
Problem: Optimal Data Placement
priorities groups pool of queries TCF monitor system
![Page 21: Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation](https://reader036.vdocuments.site/reader036/viewer/2022062317/5a4d1ae47f8b9ab0599789f1/html5/thumbnails/21.jpg)
Task
create and implement a system for an optimal data placement