usman shakeel - cloud rendering at scale :: aws rendering seminar
TRANSCRIPT
![Page 1: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/1.jpg)
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Usman Shakeel – Amazon Web Services
Cloud Rendering at scale
on AWS
![Page 2: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/2.jpg)
Visual Effects and
Animation1
Who is using AWS for Rendering?
3 Theme Parks
5 Gaming
Marketing2
4 Manufacturing
6 Life Sciences
7 Engineering and Architecture
![Page 3: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/3.jpg)
Visual Effects and
Animation1
![Page 4: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/4.jpg)
VFX/Animation Rendering - workflow components
CompostingModeling Rendering
Asset Management
Collaboration and Task Management
![Page 5: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/5.jpg)
The challenge of making a film
![Page 6: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/6.jpg)
The challenge of making a film
On-premise capacity
![Page 7: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/7.jpg)
The challenge of making a film
On-premise capacity
Rendering in the Cloud
![Page 8: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/8.jpg)
The challenge of making a film
On-premise capacity
Rendering in the CloudCloud provides you the capability to
scale fast and get the outputs faster
Initial project on-boarding
artwork
![Page 9: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/9.jpg)
Classifying Studios By Consumption
![Page 10: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/10.jpg)
They all ask us the same thing …
The ability to spin up thousands of cores on-demand
… without any upfront investment
… and leverage the most up-to-date configurations
A project based “disposable” infrastructure
… with a flexible licensing / utility / by the hour
![Page 11: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/11.jpg)
They all tell us the same thing …
=< $0.01per core/hour
Access to thousands of
cores whenever needed
No upfront investments in
infrastructure
Easier collaboration
Ecosystem of software
providers
Access to large memory
configs to do 6K/10K renders
Project based “disposable” infrastructure
![Page 12: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/12.jpg)
… when rubber meets the road !
Share FS Everywhere Latency Large Datasets Lots of instances
{Data/Content}
![Page 13: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/13.jpg)
Rendering in the Cloud
![Page 14: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/14.jpg)
Rendering in the Cloud - State of the Union
Scale at a very cheap price
EC2 Spot
![Page 15: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/15.jpg)
Leveraging Spot successfully today requires some
effort
Build stateless, distributed, scalable applications
Choose which instance types fit your workload the best
Ingest price feed data for AZs and regions
Make run time decisions on which Spot pools to launch in based on price and volatility
Manage interruptions
Monitor and manage market prices across AZs and instance types
Manage the capacity footprint in the fleet
And all of this while you don’t know where the capacity is
Serve your customers
![Page 16: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/16.jpg)
Spot Fleet
Instead of writing all that code to manage Spot Instances,
simply specify:
• Target Capacity – The number of EC2 instances that you want
in your fleet.
• Maximum Bid Price – The maximum bid price that you are
willing to pay.
• Launch Specifications – # of and types of instances, AMI id,
VPC, subnets or AZs, etc.
• IAM Fleet Role – The name of an IAM role. It must allow EC2 to
terminate instances on your behalf.
![Page 17: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/17.jpg)
Spot Fleet Example – Instance WeightingSay your workload needs at least 60 GB of memory
Want capacity to complete 20 units of work
Choices:• r3.2xlarge (61.0 GB, 8 vCPUs) = 1 unit of 20
• r3.4xlarge (122.0 GB, 16 vCPUs) = 2 units of 20
• r3.8xlarge (244.0 GB, 32 vCPUs) = 4 units of 20
An option to bid for all of these instance types:
![Page 18: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/18.jpg)
AWS Cloud Scale is “Large”
•10s/100s/1000s/10000s cores on-demand in the Cloud
•A “Large” renderfarm (Actual example):
55,000 cores
•At re:Invent 2015:
~40,000 cores on
EC2 Spot Market
Rendering in the Cloud - State of the Union
Scale at a very cheap price
![Page 19: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/19.jpg)
• BYOLVray, Solid Angle, Autodesk Maya 3DsMax
• SaaS
Thinkbox Deadline Cloud Wizard
• AWS Marketplace
• Elastic Licensing models
Thinkbox Deadline Usage Based Licensing
• Render nodes pull metered licenses from Cloud-based license server
• Usage is tracked per minute
• Bulk minutes will be available via Thinkbox’s online store
• Store will eventually host 3rd party licensing (Nuke, VRay, etc)
Pixar Renderman/Tractor (demo at SIGGRAPH)
Rendering in the Cloud - State of the UnionLicensing at Cloud Scale
![Page 20: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/20.jpg)
Rendering in the Cloud - State of the UnionHydrating the Cloud Renderfarm
S3 as the source of truth for your content/data
• On AWS Marketplace/SaaS(Aspera, Signiant, File Catalyst, Expedat)
• S3 Multi-part Upload
• AWS import/export Snowball
Direct to Shared File Systems
• EFS throughput scales linearly to the storage
• Lustre can hydrate from an S3 bucket
• Avere can be fronted to S3 or an on-premise NAS
+ Direct Connect
EFSS3
Multipart
AWS Snowball
![Page 21: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/21.jpg)
Rendering in the Cloud - State of the UnionShared FileSystem Everywhere (some ideas)
Shared Storage
On-prem StorageAWS Direct Connect
Storage Cache
Amazon S3
Luster on EC2
Avere on EC2
EFS
AWS Direct Connect
Hydrate workers
EC2 Spot
Shared Storage
FXT on-prem
![Page 22: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/22.jpg)
Rendering in the Cloud - State of the UnionShared FS (Content/Data Share) Everywhere (some ideas)
Elastic File System (Amazon EFS)
• Designed to support Peta-Byte scale file systems
• Throughput scales linearly to storage
• Same latency spec across each AZ
• Thousands of concurrent NFS connections
• Works great for Large I/O sizes
• Pay for only what you use not what you provision
• Managed with multi-copy durability Amazon EFS
![Page 23: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/23.jpg)
Rendering in the Cloud - State of the UnionMove the Graphic Artist to the Cloud …
Rendering is going Global• NVIDIA GPU based EC2 instances
• Nice DCV
• Teradici PCoIP
• Frame, Otoy
• Windows and Linux (VNC+VirtualGL)
3D Modeler
Modeling Dumb Client
Remote Application
running on a G2 instance
G2
![Page 24: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/24.jpg)
Rendering in the Cloud - State of the UnionManaging your “disposable” infrastructure
Launch a CloudFormation stack
with all the infrastructure
resources for a specific project
Autoscale the stack as
appropriate
AMI
CloudFormation
TemplateCloudFormation
Terminate
Template
![Page 25: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/25.jpg)
Rendering in the Cloud - State of the UnionThe Crown Jewels
• AWS alignment with the latest MPAA cloud based application
guidelines for content security – August 2015
• VPC private endpoint for S3 – enables a true private workflow
capability
• Encryption & key management capabilities
• Glacier Vault for high-value media/originals
![Page 26: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/26.jpg)
Rendering in the Cloud - A Sample Architecture
(All in Cloud Pipeline)
Shared Storage
Renderfarm
On-Prem Storage
Pipeline and License Manager
3D Modeler
Remote
App Visualization
AWS Direct Connect
Modeling Dumb Client
Storage Cache
Amazon S3
Avere on EC2
Scalable Renderfarm on EC2
Appstream or Teradici running on a G2 instance
Pipeline Manager running on EC2
G2
EC2 SPOT
EFS
Hydrate workers
EC2 Spot
![Page 27: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/27.jpg)
Render Farm
Rendering in the Cloud - A Sample Architecture
(A Hybrid Pipeline)
Shared Storage
Renderfarm
On-Prem Storage
AWS Direct Connect
Storage Cache
Amazon S3
Avere on EC2
Scalable Renderfarm on EC2
EFS
Hydrate workers
EC2 Spot
On-premise
RenderfarmCloud renderfarm as an
extension of on-prem renderfarm
FXT on-prem
Pipeline and License
Manager (also manage
cloud renderfarm)
![Page 28: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/28.jpg)
A Customer Example
A large scale test:
• Deploying ~40K cores
• In 20 min
• Leveraging Spot Fleet
• Hybrid rendering environment
using Avere (on-premise Data)
• EFS
Faster than on-premise
environment
![Page 29: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/29.jpg)
Disney Animation Renderfarm
RenderfarmAvere FXT
cluster
WDAS Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
RenderfarmAvere FXT
cluster
Remote Data Center
San Francisco
Los Angeles
Burbank
Artists
Redundant 10Gb
![Page 30: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/30.jpg)
Disney Animation Renderfarm (on the Cloud)
RenderfarmAvere FXT
cluster
WDAS Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
RenderfarmAvere FXT
cluster
Remote Data Center
San Francisco
Los Angeles
Burbank
Artists
Redundant 10Gb
virtual private cloud
Avere vFXT
Oregon
Spot InstancesEFS
![Page 31: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/31.jpg)
Spot Fleet Deployment
![Page 32: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/32.jpg)
Another Customer Example
![Page 33: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/33.jpg)
Cloud Rendering Benchmarks
![Page 34: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/34.jpg)
Benchmarks: On Premise vs. The Cloud
0
20
40
60
80
100
120
streamtriad diskread diskwrite
OnPrem
r3.4xlarge
r3.8xlarge
m4.4xlarge
m4.10xlarge
cr1.8xlarge
Higher is better
![Page 35: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/35.jpg)
Average Open Latency
![Page 36: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/36.jpg)
Average Read Latency
0
100
200
300
400
500
600
700
100 500 800 1200 2400 4000
Tim
e (
µs)
Render Processes
Mid-TierA
Mid-TierB
Mid-TierC
Archive
EFS
![Page 37: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/37.jpg)
Rendering in the Cloud vs. On-Premise
-
5,000
10,000
15,000
20,000
25,000
30,000
1 10 20 30 40 50 60 70 80 90
Re
nd
er
Tim
e (
s)
Frame #
EC2/EFS
OnPrem
Lower is better
![Page 38: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/38.jpg)
Lessons Learned
• Use as many different instance types as you can. Especially older generations.
• Think about ways to modify your workload
• Use every availability-zone
• Check your limits, especially your EBS limit and VPC setup (address space)
• Resource-Oriented Bidding
• Diversified Allocation
• Benchmark your workload and set pricing accordingly
• Set ONLY realistic pricing that you will pay for
• Don’t be afraid to ask for help or pre-planning your run from AWS
![Page 39: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/39.jpg)
Conclusion
Cloud Rendering on AWS
State of the Union is getting stronger …
Rendering Forecast
Partly cloudy with a chance of all in the cloud…
Technology Trends
•Containers for Rendering
•GPU based Rendering
•More vendors investing in Elastic Licensing models due to the push from their
customers
•All in-Cloud rendering with more robust application streaming solutions
![Page 40: Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar](https://reader031.vdocuments.site/reader031/viewer/2022021922/587c78111a28abd04e8b73a1/html5/thumbnails/40.jpg)
Thank you!