a personal cloud controller yuan luo school of informatics and computing, indiana university...
TRANSCRIPT
![Page 1: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/1.jpg)
A Personal Cloud Controller
Yuan Luo
School of Informatics and Computing, Indiana University Bloomington, USA
PRAGMA 26 Workshop
![Page 2: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/2.jpg)
Problems• To fully leverage existing resources for their applications, users
often need to manually configure available resources to a custom configuration, a time consuming task.
• Labs may have a suite of applications with different requirements and runtime profiles (e.g., compute intensive, data intensive, data sensitive) necessitating different scheduling approaches.
• Collaborative e-Science projects typically require data processing to be performed on distributed data sets. Some data are so sensitive that they cannot be processed out of certain administrative boundaries.
• Checkpointing is a direction to achieve fault tolerance. But in some cases, checkpointing is expensive as it consumes large amount of time and resources.
![Page 3: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/3.jpg)
Solution Space
• User Controllability• Resource Allocation• Sensitive Data Processing• Fault Tolerance for Long-Run Applications
![Page 4: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/4.jpg)
Personal Cloud Controller(Solution to user controllability)
• Personal Cloud Controller (PCC) integrates and leverages PRAGMA tools to easily deploy and manage virtual clusters.
• PCC is lightweight and extends HTCondor for scheduling and fault tolerance capabilities. PCC – Provides flexible interface to enable a high level of user
controllability,
– Automates configuration, deployment, and fault recovery on behalf of user.
Personal Cloud Controller
Rocks
PCC-HTCondor
OpenNebula …
Web Interface
PRAGMA tools (pragma_boot, ViNE, iPOP)
![Page 5: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/5.jpg)
Personal Cloud Controller – cont’dFundamentals of the following…
• Resource Allocation– Resources and applications advertising
– Match resources and applications
• Sensitive Data Processing– PCC enabled PRAGMA Cloud
• Long-Run Applications Fault Tolerance– HTCondor’s checkpointing mechanism
(complementary to provenance-based fault tolerance)
![Page 6: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/6.jpg)
Client
The PRAGMA Cloud
Cluster A
Cluster B Cluster C
Cluster D
Allocated Resource
Unclaimed Resource
Physical Network
Virtual Network
PCC-HTCondor Master
Provenance Collecting Path
Cluster Master Node
PCC Enabled PRAGMA Cloud
![Page 7: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/7.jpg)
Negotiator
Collector
Startd
Schedd
Central Manager
Startd
Schedd
Shadow
Machine 1 (submit)
Startd
Schedd
Starter
Machine N (execute)
VM GAHP
PRAGMA Cloud tools
(pragma_boot)
Communication PathProcess Invoke
…
PCC-HTCondor Architecture
![Page 8: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/8.jpg)
Ongoing and Future Work
• Develop a set of application-driven resource allocation strategies and scheduling algorithm based on data locality, sensitivity, etc.
• Extend the Hierarchical MapReduce model to support sensitive data processing.
• Develop a provenance-based fault-tolerance model to handle long-run applications.
![Page 9: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/9.jpg)
Acknowledgement
• The author would like to thank – Dr. Beth Plale (Indiana University)
– Dr. Philip Papadopoulos, and Shava Smallen (UCSD)
• This work funded in part by NSF Award OCI 1234983
![Page 10: A Personal Cloud Controller Yuan Luo School of Informatics and Computing, Indiana University Bloomington, USA PRAGMA 26 Workshop](https://reader036.vdocuments.site/reader036/viewer/2022082612/56649f4e5503460f94c6f35d/html5/thumbnails/10.jpg)
Thank you!