daaas: behind the scenes - queen mary university of london
TRANSCRIPT
![Page 1: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/1.jpg)
DAaaS: Behind the scenes
Frazer Barnsley, May 2019
![Page 2: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/2.jpg)
Overview• User facing capabilities
• Virtual Machine Manager (VMM)• VM Configuration• ISIS Cloud• External Clouds
• Data Movement Service (DMS) / Data Transfer Service (DTS)
• HPC integration
![Page 3: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/3.jpg)
User Facing Capabilities• Users can select a virtual research environment for a
specific type of analysis
• Near instant – takes a couple of seconds
• Access via SSH, RDP, NoVNC
![Page 4: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/4.jpg)
User Facing Capabilities
![Page 5: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/5.jpg)
Virtual Machine Manager• Manages the creation, deletion of VMs
• Provides a web API• Uses python libcloud• VM pools• Machine types – links to environments
• Key Value pairs associated with machine types
![Page 6: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/6.jpg)
Virtual Machine Manager
![Page 7: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/7.jpg)
Virtual Machine Manager
![Page 8: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/8.jpg)
VM Configuration• All environments based off Centos 7 Mininal
• Completely configured via Ansible• Service called ansbilecontroller• Pool management: ansible controller changes state
from Provisioned to Vacant once configured
Provisioned Vacant Acquired Deleted
Failed
![Page 9: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/9.jpg)
VM Configuration
![Page 10: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/10.jpg)
ISIS Cloud
• Another Openstack cloud• Provides redundancy and extra capacity• Add a new cloud entry into the VMM• Up to web frontend on how to present multiple clouds
to user
![Page 11: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/11.jpg)
ISIS Cloud
![Page 12: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/12.jpg)
External Clouds
• IRIS Clouds, AWS etc …• Same as adding ISIS cloud
• Issues around data access
![Page 13: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/13.jpg)
External Clouds
![Page 14: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/14.jpg)
Data Movement Service (DMS) / Data Transfer Service (DTS)
• DTS– Python service with web API
– Inspiration from GridFTP
– Allows 3rd party transfer
• DMS– FUSE client running on VMs
– Provides a mirror image of remote data store
– Intelligently caches data in background
![Page 15: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/15.jpg)
DTS
![Page 16: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/16.jpg)
DMS
![Page 17: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/17.jpg)
DMS / DTS
![Page 18: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/18.jpg)
HPC
• Looking to link DAaaS VMs to HPC
• Aspiration to make this invisible to user
• Looking to create prototype for ISIS Pace project• Pace project is linked the ISIS Excitations group (ie.
one of the production DAaaS environments)
![Page 19: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/19.jpg)
HPC
![Page 20: DAaaS: Behind the scenes - Queen Mary University of London](https://reader033.vdocuments.site/reader033/viewer/2022043006/626b418fb4b1782cda394266/html5/thumbnails/20.jpg)
Next Steps
• VMM going live in production this Thursday
• Integration of ISIS Cloud set for 10th September• HPC integration – aiming for 6 months