Synthesizing 3D Worlds

- Baskar Rethinasabapathi

Why do we need this?

- laborious task; saves designer time- Building an avg scene takes upto 20 minutes

Why do we need this?

- laborious task; saves designer time- Building an avg scene takes upto 20 minutes

- constant need for variety

Why do we need this?

- laborious task; saves designer time- Building an avg scene takes upto 20 minutes

- constant need for variety

Systems find it difficult

- High dimensional ; complex to model- Object properties differ

- Constraints based- Interactive and dependent

Existing approaches

- Procedural modeling - Should have a structural grammar

- Component based- Not suited for scenes

- Evolutionary- Repetitive; less creative

- Probabilistic- To be discussed

Potential application

– Creating indefinite virtual worlds

No man’s sky – Procedural

http://youtu.be/ZVl1Hmth3HE?t=28s

Problem domains addressed

System 1 - Example-based Synthesis of 3D Object Arrangements

- focus on local arrangements

System 2 - Synthesizing open worlds with constraints using locally annealed reversible jump MCMC

- focus on open world arrangements

Paper Source

1. http://dl.acm.org/citation.cfm?id=2366154

2. http://dl.acm.org/citation.cfm?id=2185552

High-Level Challenges

- output should be highly plausible/agreeable

- should generate a very large variety

- Users should not be involved much

General approaches

1. Example based

2. Constraints based

Dataset

- Objects created - from Google 3D warehouse

- by users and annotated

Goal of System 1

- Users should judge synthesized scenes to be highly plausible compared to hand created ones

- At least one out of every three synthesized results should be usable

System design 1

Scene -> Contextual + User example -> Learneddatabase Categories Input scenes Mixed Model

Probabilistic Mixed Model =

Occurrence model + Arrangement model

Goal of System 2

- Should be more applicable on open world layouts

- Optimal object arrangements which is diverse enough

System design 2

Scene database ->Extract system of constraints -> Generate Factor graphs

Repeated constraints in Scene -> novel MCMC method -> new scenes

System 1 - Algorithm

- Build a bipartite matching graph between objects in image

- Edge cost proportional to = same basic category+ smaller distance

- Max weight = Scene’s static support + Size of Objects + Edge cost

Clustering from the database

- Randomly choose anchor object – initialize a basic cluster

- Align with = neighborhood similarity + geometric features

- Merge clusters

Interesting sub problems

System 1

- Finding contextual neighborhood- How do we find a cookie on a plate in image A is interchangeable with bread

on image B?- Look for forks and knives around

- Similar in type and arrangement

- Finding hierarchical static support- Cookie should be on a plate, not on the bed

- Filling gaps in user examples- Construct a Bayesian network from database

System 2 - AlgorithmSystem 2

- Defining constraints (π1)- numPlates = randInt(1,10)

- Size_Table = Uniform(15,25)

- Position_Plates = uniform(-10,10),uniform(-10,10)

- Size_Plate = uniform(0.1,2)

- Building factor graphs- 70% of the table should be occupied (π2)

- softEq(0.7, total_area(plates)/ area(table))

- Plates should be inside the table (π3)- For all i, softEq(0.0, area_outside_table(plate_i))

- Total constraints model = π1* π2* π3

System 2 - AlgorithmSystem 2

- Markov Chain- Next state is dependent only on present state

- Monte Carlo Markov Chain- The system of Markov chains attains an equilibrium eventually

- Reversible Jump MCMC- Should accommodate trans-dimensional constraints

- Locally annealed Reversible Jump MCMC- Do not accept/reject new constraints immediately

Demo

Analogy with other papers

Bricolage - example based learning

Discussion

- How usable are the scenes generated from both systems?

- Why can’t System 1 be scaled to open world layouts?Or, vice versa. Can you apply System 2 for local arrangements?

Discussion

- Consider the Coffee Shop Layout. If I want all kinds of fabric in the scene to be changed to leather, how do I achieve?

- Which of the two systems is more suitable for this problem?

Results

- Yes. Goals achieved

- output should be highly plausible/agreeable- Should generate a very large variety- Users should not be involved much

- Users agreed with synthesized results of System 1

- No evaluation in System 2- Focus on optimization compared to other greedy algorithms

Critique