GPU Renderfarm with Integrated Asset Management & Production System (AMPS)

Tackling two main challenges in CG movie production

Multi-plAtform Game Innovation Centre (MAGIC), Nanyang Technological University (NTU), Singapore

Presenter: Dr. Qiu Jie

Two main challenges in CG movie production

• How to manage the rendering assets efficiently

– Large amount of digital assets

– Huge load of resources and common outsourcing practice

• How to perform CG rendering in a relatively shorter time

– CPU renderfarm systems usually require substantial hardware setup, high maintenance cost, and

power consumption

• Not affordable for small studios

– Rendering algorithms can benefit from parallelization

• GPU renderfarm has more cores, smaller size, and lower cost comparing to CPU one

2

• AMPS (Asset Management & Production System)

– Online resource management (e.g. for archiving, retrieving, tracking, without location constraint)

– Can be adopted for other industries

• Integration of GPU renderfarm with AMPS

– Ability to manage the rendering assets efficiently

– Streamlining the rendering pipeline (i.e. sending assets to the renderfarm online via web browser)

– Lower rendering cost and faster rendering process

Our solutions

3

• Not flexible to satisfy the needs in all or various media production companies and

studios (e.g. Shotgun and TACTIC)

• Lack of resource management features such as revision control, workflow management,

flexible accessing / sharing rights management (e.g. Dropbox, Box.net)

• No linkage with rendering solutions

Existing asset and workflow management solutions

4

• CPU-based renderfarms

– GreenButton (RenderMan Pro Server, YafaRay, LuxRender)

• Provides external software to upload assets, monitor progress, download results, but does not have complete asset management

features

• GPU-based renderfarms

– Render Street, renderFlow, ultrarender (Cycles and/or iRay renderers)

• Rendering jobs are done in the company servers

– Sheep it! Render Farm (Cycles)

• Rendering job is distributed among the clients/users

– OTOY (Octane)

• Utilizes NVIDIA GRID

Existing renderfarm services

5

Proposed workflow

6

3D scenes & other

data

Render request

3D scenes &

Render request

Rendered results

Rendered results &

other data

HP SL250s Gen 8 * 2NVIDIA TESLA K20X * 6

• Back-end

– For storing data (e.g. projects, assets, users,

workflows, etc.)

– Provides back-end functions

– Deployed as a combination of web services and

MVC (Model-View-Controller) infrastructure

• Front-end

– Provides a user-friendly interface to access and

manipulate assets

– In various flavors : web-based (internet

browser), PC client apps, mobile client apps

Asset management component

7

Web-based Front-End

8

Native client apps

9

• Asset

– Media asset in production (e.g. images, videos, sounds, 3d models, etc.)

– Version control support (i.e. revisions are kept for each asset)

– Can be shared to internal and/or external users

• User

– Each user has roles (that can be created/deleted/edited and assigned to different projects)

– Two types, real user or virtual user

• Real user : regular user, such as artist, director etc.

• Virtual user : interface between AMPS and external application / solution e.g. renderfarm.

AMPS Components

10

• Workflow

– Production tracking feature

– Consists of workflow Steps

– Multiple workflow support

– Assets can be assigned to steps or transferred to another workflow

• Project

– Typically corresponds to a real-life project such as an animation, film, or game project

– Consists of a set of assets, asset revisions, asset folders, project roles, and users etc.

AMPS Components

11

Request asset info (3)

Download assets request (5)

Renderfarm component

12 Web browser

Rendering

Request (1)

AMPS

Thin app

Backburner

Manager

Send job

(8)

Trigger & inputs (2)

Assets (6)

Job &

Assets (10)

Assets

(7)

Assets

(9)

Job &

Assets (10)

Job &

Assets (10)

Results (12)

Results (13)

Rendering results (14)

Results (11)

Results (11)

Results

(11)

Asset info (4)

Manager Node Rendering Nodes

• Consists of a manager node and a set of rendering nodes

• Manager node consists of :

– Thin app : our in-house developed software, to facilitate communication between AMPS and the renderfarm

– BackBurner Manager : sending assets to rendering nodes and distributing rendering tasks

• Each rendering node consists of :

– BackBurner Server : to receive assets and rendering jobs from manager node, and to return the rendering

results back to the manager node

– Rendering software : software to perform rendering

Renderfarm component

13

• Serve as the middleman between AMPs and the rendering nodes

• Responsibilities :

– Awaiting and receiving rendering requests from AMPS

– Requesting additional information supporting rendering assets (such as textures) from AMPS

– Downloading rendering assets from AMPS to a temporary folder

– Sending rendering jobs to the Backburner Manager

– Awaiting and monitoring a temporary folder for the rendering results

– Uploading the rendering results to AMPS

Thin App

14

• Rendering job submission is received by Thin App through the network (local or internet) using TCP

protocol

• The job submission message is sent in JSON format, comprises :

– Action type (“exec function”) : specifies what action to do

– Array of 3D scene files to be rendered (“data array”) : extendable for submitting multiple rendering job

simultaneously

– Each object (“item”) in the array corresponds to one rendering job. It consists of :

• Information related to the main 3D scene file (e.g. its Asset ID in AMPS database, file size, file name ,etc.)

• Asset ID of each dependent asset (e.g. textures, mocap file, etc.)

• Rendering parameters (e.g. output resolution, output filename, folder ID in AMPS to store the output, etc.)

Thin App

15

Rendering job submission

16

• The system is used in the Asia’s first live-action Mecha feature film “The Boy And His Robot”,

the first fully GPU (Graphics Processing Unit) rendered feature film in post-production

• Experiment scenario

– Artists from various places (e.g. Singapore, China, Russia, France etc.) create and upload assets to

AMPS, managing assets using AMPS

– Movie director in Singapore evaluates their works through AMPS

– Revisions and final approval from the director

– The 3D assets can then be submitted to the renderfarm

Experiment

17

• Rendering specifications :

– Resolution : 1920 x 1080

– Polygons : 752,608

– Frames : 85

– GPU RAM consumption : 2,814 MB

– Rendering : Path Tracing, 1,250 samples per pixels

and maximum 3 bounces

• Manager node specifications :

– OS : Windows 7 Enterprise

– CPU : Intel Core i7 920 2.67 GHz & 2.67 GHz

– RAM : 6 GB

Rendering experiment

18

• Rendering node specifications :

– OS : Windows Server 2012

– Renderer : OctaneRender Autodesk 3DS Max Plug-in

(ver. 1.18), is hosted in Autodesk 3DS Max 2014

– CPU : Intel Xeon E5-2660 2.20 GHz & 2.19 GHz

– RAM : 64 GB

– GPU : 3x Tesla K20X

Hardware ConfigurationRendering

Time

One node, with one K20X GPU 5:58:45

One node, with two K20X GPUs 3:11:46

One node, with three K20X GPUs 2:15:34

Two nodes, each with one K20X GPU 3:00:24

Two nodes, each with two K20X GPUs 1:36:37

Two nodes, each with three K20X GPUs 1:08:47

Rendering Experiment

19

• Rendering speed up for one node :

– 1 GPU -> 2 GPUs : 1.87x (~2.0x)

– 1 GPUs -> 3 GPUs : 2.65x (~3.0x)

– Similar trend for two nodes

• Rendering time for two nodes vs one (each node with the same # GPUs) is almost 0.5 : 1

– Currently one frame can be handled by only one GPU node

• Total acceleration : 1 node with 1 GPU only -> 2 nodes with 3 GPUs per node : 5.22x

Rendering experiment

20

• Conclusion

– Our system can aid movie production pipeline in term of efficiency and saving rendering cost

– We can obtain rendering time acceleration that scales almost linearly with the number of GPU nodes

• Future Work

– AMPS plug-in for 3D authoring tool (e.g. 3DS Max) so that the rendering job can be submitted directly

from the authoring tool

– Support for other 3D authoring tools (e.g. Maya and Blender) and other rendering software

– Support for heterogeneous GPU renderfarm (rendering nodes with different OS, brand of GPUs, and

maybe geographically separated)

Conclusion and Future Work

21

• Singapore National Research Foundation under its IDM Futures Funding Initiative and

administered by the Interactive & Digital Media Programme Office, Media Development

Authority

• Ministry of Education Singapore for the Tier-2 research funding support

• HP and NVIDIA for providing server nodes and GPUs

• Richmanclub Studios for the 3D models, test-bedding activities and feedbacks

• OTOY for providing the OctaneRender

Acknowledgements

22

Demonstration & Discussion