Image Mix and Match

• Internet images– Colorization– Most similar image searching– Collage

• Object insertion

• Data-driven approach for robust similarity measure– Cross domain(Photo, Photo with different lighting, Painting)– No domain specific treatments

• 異種画像に利用できる画像の類似度計算法

• Idea– Detect unique region of the target image (comparing to the other)– Place high weight on the unique regions– 与えられた画像のどの特徴が， Web 上の膨大な量の画像に対してユニークかを学習– ユニークな画像特徴に重みを置く

• Image feature vector ( 画像特徴ベクトル )– Intensity histogram, Gradient Magnitude Histogram, HoG, SIFT– 画像間距離は，この特徴ベクトル間の距離として計算する事が多い

Intensity histogramHoG, Histogram of oriented gradient ， 4k-5k dimension

• Linear support vector machine (see Pattern recognition textbook)• Given d-dimensional feature vectors belongs to class A and B, xi A, ∊ yi B ∊ xi, yi R∊ d

Find the maximum-margin hyperplane that divides xi A, ∊ yi B∊

• この絵は２Ｄだけど本当は特徴ベクトルと同じだけの次元，5000 次元とか

xi

yi

Goal• Given a image Ip

– Detect unique parts of feature vector of Ip 　 comparing to the others– Place high weight on the unique regions

Ip

Other images on Web

1) Compute feature vectors of Ip xp and the other internet images xi

2) Compute hiperplane by linear SVM3) Project feature vectors of all images onto the

normal of the hiperplane

この normal が PCA の軸のような役割になる

• Image colorization from internet image• Gray scale 画像の色付けを， Web 上の画像を参照して行う– Combination of many techniques• Internet image search, foreground segmentation, suitable

image filtering, Image similarity measure, graph-based color transfer, selection UI for weight tuning

• Input: Image + text label (e.g. rooster)

• Procedure– Input: grayscale image with foreground segmentation & text label

– Search images from internet• Google image search / Flickr• Automatic fore ground extraction• Filter similar images for back/fore ground(Ad hoc energy function intensity, texture,

density of SIFT)

– Color transfer• Graph based color transfer method

– Maintain the neighborhood consistency• Compute with different weighting values The user can select one of them

– Output: Colored images with different reference images and different weighting

values

Color transfter の計算時に， gray 画像と参照画像の両方とも，微小領域に分割し，近傍に矛盾がないような color transfer を計算している．例えば蝶の羽などでは，黄色の隣にはオレンジ色が来る事は無いなど，良い結果を生んでいる．

• Arcimboldo-style collage generation– Input : Source image and text label for searching element image cutouts– Output: collage consists from internet images

• Giuseppe Arcimboldo 1527-1593• Itary• Collage like drawing

– Each element is recognizable (elements are taken from a same theme)– The assembly of the elements resembles something

Two problems• Best fitting cutout search • Input image segmentation

1) Mean shift clusteringCompute modes in color & space feature space エッジ保存フィルタを連続して書けるような物2) Marge & split strange local regions-Mean shift clustering generates local regions that not match any element cutout image from internet-Trim such regions by Ad hoc iteration

1) Search images from internet2) Cutout foreground image by saliency detection and GrabCut3) Distance metric between - Hole in the target image - Cutout image from internet

Color distance term / Shape distance termWith best fitting affine transformation

Semantic aware segmentation is difficult…

• New tool for inserting objects into Photographs– geometry and light estimation with user’s guide

• Less user interaction

– Deal with interior light and exterior lights (e.g. sun light from window)– ある点がすごい新しいとかではなく，他と比較して全体的なパフォーマンスが上がっている感じ

• Inputs– Single image– User annotation (geometry, light source position)– 3D model that will be inserted into the Image

Overview of the system

• Geometry estimation– Previous work + user’s correction, user interface to add other geometry

• Light source estimation– Next page

• Object insertion– Add object into 3D scene and render it with estimated parameter

Interior light estimation

1) Decompose input image into Albedo and direct light image2) User points the position of the light3) Automatically adjust light parameter (position & RGB)!!Full automatic light source detection from single image is very difficult!!

= +

Exterior light estimation

1)The user marks boundary of the source and projection of the exterior light2) The system automatically computes mask and direction

• 今年の SIGGRAPH に• Image = Albedo + Direct• Image = Albedo + Indirect + Direct という分解をする論文あった

Results