putting eggs in many baskets: data considerations in the cloud rob futrick, cto
TRANSCRIPT
![Page 1: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/1.jpg)
Putting Eggs in Many Baskets:Data Considerations in the Cloud
Rob Futrick, CTO
![Page 2: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/2.jpg)
We believe utility access to technical computing power accelerates discovery & invention
![Page 3: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/3.jpg)
The Innovation Bottleneck:
Scientists/Engineers forced to size their work to the
infrastructure their organization bought
![Page 4: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/4.jpg)
Limitations of fixed infrastructureToo small when needed most,
Too large every other time…
• Upfront CapEx anchors you to aging servers
• Costly administration
• Miss opportunities to do better risk management, product design, science, engineering
![Page 5: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/5.jpg)
Our mission: Write software to make
utility technical computingeasy for anyone,
on any resources, at any scale
![Page 6: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/6.jpg)
As an example…
![Page 7: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/7.jpg)
Many users use 40 - 4000 cores, but let’s talk about an example:
World’s first PetaFLOPS ((Rmax+Rpeak)/2)
Throughput Cloud Cluster
![Page 8: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/8.jpg)
![Page 9: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/9.jpg)
What do you think?• Much of the worlds
“bread basket” land will be hotter and drier
• Ocean warming is decreasing fish populations / catches
![Page 10: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/10.jpg)
First, buy land in Canada?
![Page 11: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/11.jpg)
Sure! But there have to be engineer-able solutions too.
Wind power
Nuclear Fusion
GeoThermal
Climate Engineering
Nuclear Fission energy
Solar Energy
BioFuels
![Page 12: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/12.jpg)
Designing Solar MaterialsThe challenge is efficiency - turning photons to electricity
The number of possible materials is limitless:• Need to separate the right compounds from the useless ones• Researcher Mark Thompson, PhD:
“If the 20th century was the century of silicon, the 21st will be all organic. Problem is, how do we find the right material without spending the entire 21st century looking for it?”
![Page 13: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/13.jpg)
Needle in a Haystack Challenge:
205,000 compounds totaling 2,312,959 core-hours
or 264 core-years
![Page 14: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/14.jpg)
16,788 Spot Instances, 156,314 cores
205,000 molecules264 years of computing
![Page 15: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/15.jpg)
156,314 cores = 1.21 PetaFLOPS (~Rpeak)
205,000 molecules264 years of computing
![Page 16: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/16.jpg)
8-Region Deployment
US-West-1US-East
EU
US-West-2
BrazilSingapore
Tokyo
Australia
![Page 17: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/17.jpg)
1.21 PetaFLOPS (Rmax+Rpeak)/2, 156,314 cores
![Page 18: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/18.jpg)
Each individual task was MPI, using a single, entire machine
![Page 19: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/19.jpg)
Benchmark individual machines
![Page 20: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/20.jpg)
Done in 18 hoursAccess to $68M system
for $33k
205,000 molecules264 years of computing
![Page 21: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/21.jpg)
How did we do this?
Auto-scalingExecute Nodes
Auto-scalingExecute Nodes
JUPITER Distributed
Queue
JUPITER Distributed
Queue
DataData
Automated in 8 Cloud Regions, 5 continents, double resiliency
…14 nodes controlling 16,788
![Page 22: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/22.jpg)
Now Dr. Mark Thompson• Is 264 compute
years closer to making efficient solar a reality using organic semiconductors
![Page 23: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/23.jpg)
Important?
![Page 24: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/24.jpg)
Not to me anymore ;)
![Page 25: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/25.jpg)
We see across all workloads
Interconnect Sensitive
Hi I/O Big Data runs
Large Grid MPI Runs
Needle in a Haystack runs
Whole sample set analysis
Large Memory
Interactive, SOA
![Page 26: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/26.jpg)
Users want to decrease Lean manufacturing’s ‘Cycle time’
= (prep time + queue time + run time)
![Page 27: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/27.jpg)
SLED/PS Insurance Financial Services Life Sciences Manufacturing & Electronics
Energy, Media & Other
Everyone we work with faces this problem
![Page 28: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/28.jpg)
External resources(Open Science Grid, cloud)
offer a solution!
![Page 29: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/29.jpg)
How do we get there?
![Page 30: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/30.jpg)
In particular how do we deal with data in our workflows?
![Page 31: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/31.jpg)
Several of the options…• Local disk• Object Store / Simple Storage Service (S3)• Shared FS
– NFS– Parallel file system (Lustre, Gluster)
• NoSQL DB
![Page 32: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/32.jpg)
Let’s do this through examples
![Page 33: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/33.jpg)
Compendia BioSciences (Local Disk)
![Page 34: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/34.jpg)
The Cancer Genome Atlas
![Page 35: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/35.jpg)
G
![Page 36: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/36.jpg)
G
![Page 37: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/37.jpg)
As described at TriMed Conference• Stream data out of TCGA
into S3 & Local machines (concurrently on 8k cores)
• Run analysis and then place results in S3
• No Shared FS, but all nodes are up for a long time downloading (8000 cores * xfer)
![Page 38: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/38.jpg)
G
![Page 39: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/39.jpg)
G
![Page 40: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/40.jpg)
Pros & Cons of Local Disk Pros:• Typically no application
changes
• Data encryption is easy; No shared key management
• Highest speed access to data once local
Cons:• Performance cost/impact of all
nodes downloading data
• Only works for completely mutually exclusive workflows
• Possibly more expensive; Transferring to a shared filer and then running may be cheaper
![Page 41: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/41.jpg)
Novartis(S3 sandboxes)
![Page 42: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/42.jpg)
(W.H.O./Globocan 2008)
![Page 43: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/43.jpg)
Every day is crucial and costly
![Page 44: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/44.jpg)
Challenge: Novartis run 341,700 hours of dockingagainst a cancer target; impossible to do in-house.
![Page 45: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/45.jpg)
Most Recent Utility Supercomputer server count:
![Page 46: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/46.jpg)
AWS Console view(the only part that rendered):
![Page 47: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/47.jpg)
Cycle Computing’s cluster view:
![Page 48: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/48.jpg)
Metric Count
Compute Hours of Science 341,700 hours
Compute Days of Science 14,238 days
Compute Years of Science 39 years
AWS Instance Count 10,600 instances
CycleCloud, HTCondor, & Cloudfinished impossible workload,
$44 Million in servers, …
![Page 49: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/49.jpg)
39 years of drug design in 11 hours on 10,600 servers for < $4,372
![Page 50: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/50.jpg)
Does this lead to new drugs?
![Page 51: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/51.jpg)
Novartis announced 3 new compounds going
into screening based upon this 1 run
![Page 52: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/52.jpg)
Pros & Cons of S3Pros:• Parallel scalability; high
throughput access to data
• Only bottlenecks occur on S3 access
• Can easily and greatly increase capacity
Cons:• Only works for completely
mutually exclusive workflows
• Native S3 access requires application changes
• Non-native S3 access can be unstable
• Latency can affect performance
![Page 53: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/53.jpg)
Johnson & Johnson (Single node NFS)
![Page 54: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/54.jpg)
JnJ @ AWS re:Invent 2012
![Page 55: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/55.jpg)
JnJ Burst use caseInternal & external
File SystemFile System
Internal Cluster
CFD/Genomics/etc.
CFD/Genomics/etc.
Submission APIsSubmission APIs
Cloud FilerCloud Filer
GlacierGlacier
Auto-scalingSecure
environment
Auto-scalingSecure
environment
HPCCluster
HPCCluster
Scheduled
Data
(Patents Pending)
![Page 56: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/56.jpg)
Pros & Cons of NFSPros:• Typically no application changes
• Cheapest at small scale
• Easy to encrypt data
• Performance great at (small) scale and/or under some access patterns
• Great platform support
Cons:• Filer can easily become
performance bottleneck
• Not easily expandable
• Not fault tolerant; Single point of failure
• Backup and recovery
![Page 57: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/57.jpg)
Parallel Filesystem(Gluster, Lustre)
![Page 58: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/58.jpg)
(http://wiki.lustre.org)
![Page 59: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/59.jpg)
Pros & Cons of Parallel FSPros:• Easily expand capacity
• Read performance scalability
• Data integrity
Cons:• Greatly increased administration
complexity
• Performance for “small” files can be atrocious
• Poor platform support
• Data integrity and backup
• Still has single points of failure
![Page 60: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/60.jpg)
NoSQL DB(Cassandra, MongoDB)
![Page 61: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/61.jpg)
(http://strata.oreilly.com/2012/02/nosql-non-relational-database.html)
![Page 62: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/62.jpg)
Pros & Cons of NoSQL DBPros:• Best performance for
appropriate data sets
• Data backup and integrity
• Good platform support
Cons:• Only appropriate for certain
data sets and access patterns
• Requires application changes
• Application developer and Administration complexity
• Potential single point of failure
![Page 63: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/63.jpg)
That’s a survey of different workloads
Interconnect Sensitive
Hi I/O Big Data runs
Large Grid MPI Runs
Needle in a Haystack runs
Whole sample set analysis
Large Memory
Interactive, SOA
![Page 64: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/64.jpg)
Depending upon your use case
File System (PBs)File System (PBs)
Faster interconnect capability machine
Jobs & dataJobs & data
Blob data (S3)
Blob data (S3)
Cloud FilerCloud Filer
GlacierGlacier
Auto-scalingexternal
environment
Auto-scalingexternal
environment
HPC Cluster
HPC Cluster
Internal TC
Blob data (S3)
Blob data (S3)
Cloud FilerCloud Filer
GlacierGlacier
Auto-scalingexternal
environment
Auto-scalingexternal
environment
HPC Cluster
HPC ClusterBlob data Blob data
Cloud FilerCloud Filer
Cold StorageCold Storage
Auto-scalingexternal
environment
Auto-scalingexternal
environment
TCCluster
TCCluster
Scheduled
Data
![Page 65: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/65.jpg)
• Approximately 600-core MPI workloads run in Cloud• Ran workloads in months rather than years• Introduction to production in 3 weeks
Nuclear engineering Utilities / Energy
• 156,000-core utility supercomputer in the Cloud• Used $68M in servers for 18 hours for $33,000• Simulated 205,000 materials (264 years) in 18 hours
Life Sciences
There are a lot more examples…
• Enable end user on-demand access to 1000s of cores• Avoid cost of buying new servers• Accelerated science and CAD/CAM process
Manufacturing & Design
Asset & Liability Modeling (Insurance / Risk Mgmt)
• Completes monthly/quarterly runs 5-10x faster• Use 1600 cores in the cloud to shorten elapsed run time
from ~10 days to ~ 1-2 days
• 39.5 years of drug compound computations in 9 hours, at a total cost of $4,372
• 10,000 server cluster seamlessly spanned US/EU regions
• Advanced 3 new otherwise unknown compounds in wet lab
Life Sciences
• Moving HPC workloads to cloud for burst capacity• Amazon GovCloud, Security, and key management• Up to 1000s of cores for burst / agility
Rocket Design & Simulation
![Page 66: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/66.jpg)
Hopefully you see various data placement options
![Page 67: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/67.jpg)
Each have pros and cons…• Local disk• Simple Storage Service (S3)• Shared FS
– NFS– Parallel file system (Lustre, Gluster)
• NoSQL DB
![Page 68: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/68.jpg)
We write software to do this…Cycle easily orchestrates workloads and data access to local and Cloud TC Scales from 100 - 100,000’s of cores Handles errors, reliability Schedules data movement Secures, encrypts and audits Provides reporting and chargeback Automates spot bidding Supports Enterprise operations
![Page 69: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/69.jpg)
Does this resonate with you?We’re hiring like crazy: Software developers, HPC engineers, devops, sales, etc.
![Page 70: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/70.jpg)
Now hopefully…
![Page 71: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/71.jpg)
You’ll keep these tools in mind
![Page 72: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/72.jpg)
as you use HTCondor
![Page 73: Putting Eggs in Many Baskets: Data Considerations in the Cloud Rob Futrick, CTO](https://reader035.vdocuments.site/reader035/viewer/2022062620/551a6503550346545e8b5c92/html5/thumbnails/73.jpg)
to accelerate your science!