Furniture Fabric Detection and Overlay using Augmented reality

Swami Balasubramanian , Dept. of Electrical Engineering, Stanford University Palo Alto, CA

Abstract— In this paper, fabric detection of furniture is investigated for the purpose of visualizing it with an augmented reality fabric of a different kind. Two segmentation algorithms using color histogram matching and color transformed MSER, is discussed and used to isolate the furniture structure from a fairly cluttered background. After isolation, filtering by image subtraction from a Laplacian of Gaussian is done to highlight the sub-sections of the furniture with the purpose of reducing fabric texture noise and shadow characteristics. After edge detection and sub-section segmentation, each of those isolated structures are augmented with another fabric type roughly retaining the orientation of the original fabric. Methods to obtain close to correct orientations are discussed. Many of the advantages and drawbacks of the methods used are discussed throughout the paper.

Keywords – Image Segmentation, Color Histogram Matching, Maximally-Stable-Extermal-Region (MSER), LoG filter, Augmented Reality, Smartphone App.

I. INTRODUCTION Many high-end furniture retails sell furniture that has

vibrant fabrics. Usually the display furniture in those stores is of a certain fabric type. Consumers visiting these stores are often given a choice of fabric materials that would be custom stitched on the furniture they choose. When the fabric selection choices run to thousands in number, customers usually face the dilemma of not being able to choose from the vast collection and visualize the fabric on the furniture.

An augmented reality application that is discussed in this paper attempts makes it possible for anyone to take a picture of the furniture in a retail store, and visualize it with the fabric of their choice.

A. Object of Interest in a cluttered background: To extract the details of the furniture, which we refer

to as the object of interest, good image segmentation needs to be done first to isolate it. There are several ways to accurately isolate the objects of interest to extract features needed for augmentation. In a typical showroom, there is no uniform background around the objects of interest and segmentation is usually a challenge. Clutter around the object includes other fabrics and furniture in close proximity.

Many image segmentation techniques are discussed in literature [1][2][3] that deal with fairly complex scenarios. The problem is typically that of image segmentation and contour detection. Using gray scale to

uniquely identify objects of interest is generally not feasible. For furniture with fabrics that have distinct colors and patterns, color segmentation works for a vast majority of cases if the furniture has uniform color patterns on the fabric. This is one of the approaches that is used here to isolate the object. A broader approach will involve integrating luminance, color and texture in clustering for feature-based segmentation as discussed in [1].

Once the segmentation is done, the object is further isolated to extract detailed features such as lines and folds that separate different geometric regions. Identifying these regions is absolutely essential if the new fabric needs to replace the original one, and be viewed in a similar manner. In some cases, it is difficult to isolate the shadow from fabric design pattern. We will discuss a technique to improve signal to noise ratio in these type of cases, where edge detection alone is insufficient to isolate regions of interest.

Once the object of interest is isolated and further regioned, each of the regions is charactereized for correct orientation. This can be a fairly complex task if orientation of the furniture relative to the camera is arbitrary. Also, certain parts of the furniture could be significantly curved. For simplicity we assume simple structures for identification, and for a proof of concept we investigate the use of very simple geometry structures such as centroids, extended intersection points of edges as a starting points. Then with the help of a reference model a perspective transformation matrix is generated, to orient the fabric correctly in each sub-regions.

Figure 1 Typical object scenario where the object of interest is the gray couch.

B. Mobile Application: These techniques need to work in a mobile app

environment. Robustness of these techniques and improvements are discussed in later sections.

One of the assumptions that is made to simplify initial segmentation is that the smartphone camera has a small focus square region that is completely enclosed by the object of interest. This helps identify the segmented region.

II. OBJECT OF INTEREST SEGMENTATION There are many techniques for image segmentation and is

one of the most researched topics in image processing and computer vision. Different situations have their uniqueness and require different techniques. Fabrics come in different types; some are heavily textured, while some are just plain without significant texture, and some have strong color characteristics.

In this implementation we use color segmentation as the method to isolate the object. Converting from RGB to L*a*b space enhances the uniqueness of the color relative to its immediate background. We use two main techniques namely color histogram matching and MSER on a color converted space.

A. Segmentation using color converted space with MSER. This technique works well for objects that have very unique color relative to its immediate surroundings, irrespective of luminance background. In some cases one of the two color spaces is dominant and an MSER technique can be used to region and isolate the object. The technique uses RGB to L*a*b conversion and uses the most dominant of the two color space to isolate the object. In this application the MSER from vlfeat.org [5] is used for segmentation since it yields the best boundary contour. The dominant centroid is chosen as the correct region of interest. The segmented images are further process to eliminate unwanted edges and geometries. The following diagram shows the results of segmentation.

Figure 2 Segmentation using Color space conversion and MSER

B. Color histogram matching This is a standard technique used [4] for comparing a

cluster of pixels if it belongs to a group of know cluster. We convert the RGB space to L*a*b and use only the a and b component to build a histogram reference. This reference is made on an image sample within a focused region of the furniture. As mentioned in the introduction mobile application section, the requirement here is that the camera view has a small square or rectangular region that is focused on the furniture, and is completely enclosed by the furniture fabric in view. This sample space is characterized and a histogram is built. This is compared with all the pixels (in a sliding window fashion) and a least distance qualifier is used to pick the correct pixels. As with all averaging statistical characteristics, edges are not uniquely defined and the segmentation is sensitive on the window geometry (pixels around the pixel under consideration). This method yields better results for complex color structures that MSER with color space conversion does not. However it needs to be used in conjunction with closed contour detection for proper partitioning of regions. This method is still under investigation.

III. FINE REGIONING OF THE OBJECT OF INTEREST A. Filtering and Fine segmenting the isolated object of

interest.

The next step involves regioning the isolated object to identify its characteristic planes and structure. Gray scale image of the isolated objects shown in Fig 2 is used as a starting point. Doing simple edge detection is significantly noise prone due to the fine structure of folds and strong characteristics of texture and shadows. Texture filtering and segmentation as described in [6][7] can used to smooth out strong texture characteristics, but does not work for cases that have significant shadows in a dark fabric.

A simple Gaussian filter is sufficient in the test cases we have adopted, to smooth out most of the texture structures. However the issue, of poor signal to noise ratio (texture and shadow noise) was still persistent by just a Gaussian filtering, and varied considerably for different images. The shadow regions widened due to Gaussian and false regions were detected that made it difficult to uniquely identify regions of interest. Canny or Sobel edge detection introduces fine lines but still picks texture and shadow boundaries.

LoG (Laplacian of Gaussian) filter subtracted from a Gamma reduction of the image worked well with images with wide boundaries. The technique consists of first filtering the image with a LoG kernel, that is fairly larger than typical shadow corners and edges. LoG is fairly in sensitive to shadow boundaries and picks the center regions of a wide shadow boundary as a segment. This is then subtracted from a Gamma reduced image (that has brightened uniformly to reduce shadows). This results in enhanced edges within the images, which then goes through a LoG filter for further regioning.

I = Im ^ 0.4 – S*LoG(Im) ; (1)

Here Im is the input gray-scale image, LoG is Laplacian of Gaussian, where S is an appropriate scale factor. The choice of

S depends on the magnitude of texture noise, and is chosen to just suppress texture noise while keeping the edges intact. Scale factor S can be chosen by finding the average intensity after filtering with LoG and eliminating the outer edge regions (which are typically much more intense). S factor is sensitive to texture and does not work for all texture type, especially ones that have long thick lines and curves. For complex textured images, other geometric based heuristic techniques need to be used for finding the best region matches, since regioning using this technique will have plenty of false regions that are difficult to isolate.

Figure 3 Edge enhancement. (Gray couch from Fig1)

Once the edges are detected using LoG on the above result, a logical image is created by thresholding. The regions have mostly well-defined closed contour regions with some disconnect in some places. This is highly dependent on lighting condition under which the picture is taken, and also the orientation of the furniture. Morphological structuring is used to close fine gaps. The edge enhancement process proved to be a vey critical step is getting properly defined regions.

Watershed segmentation is applied to label the isolated sub-regions and is used for further processing the regions. The advantage of using Watershed over conventional closed contour regioning is that thicker wall regions are shrunk to a small single pixel width.

Figure 4 Edge detected and closed gaps after Morphological processing . Partitioned for final fabric overlay.

IV. RESULT : AUGMENTED REALITY

A. Region identification : Calculating their centroids identifies regions within the segmented images. These centroids are then matched to the centroid of a reference model. An initial guess is made using the upper most central centroid, and distances calculated for each with all others. The minimum usually gives the correct match. The draw back here is that the perspective of the object is limited to visible regions only, even though a complete 3D model is available with all object plane. The algorithm can be improved further by calculating the perspective transformation, using

inclination of the lower most edges. This process is largely scale and shift in-variant. The process above identifies the planes of the sub-regions, and then a projective transformation is applied to the fabric image to be attached to each of the region. The transformation matrix is obtained in several steps. First the region under consideration is isolated, and edges detected (contours) and Hough transform is performed to detect lines along the edges. Each line is extrapolated to find the intersection points, which forms the basis of the transformation matrix T. T is calculated comparing an assumed birds view model of the identified plane quadrilateral with the new points obtained above. This gives a fairly close orientation of the fabric. In this implementation, only the Back, Seat and the Leg planes are modeled. Modeling hand rest is more complicated since it has curved surfaces, and is a subject for future improvements.

B. Color and Lighting: A very simple lighting model is used for recovering shadows and reasonably good luminance matching in appropriate regions. The gray scale image is passed through a Gaussian filter to remove texture, and this is scaled with the Luminance component of the fabric. The fabric itself is operated upon in the HSV space, so only the Value component needed to be scaled and adjusted. In general lighting value need to be obtained by comparing the luminance averages of the original image and augmented image, since is a significant reduction luminance by normal scaling (multiplying). Also as future improvements, color compensation needs to be considered to match the ambient light environment accurately. If the augmented fabric original lighting spectrum is known, the color could be compensated if a stores typical light ambience color is available as part of the mobile application.

C. Results

Results are shown at the Appendix section. Matlab was used extensively to do all the computation. There are a few observations. The contour in the main segmentation is not very accurate, and it is a challenge to get very accurate contour without noise. But the fabric very closely fits the original image despite its minor inaccuracy in contouring. A little more mask dilation would help in completely covering the original image.

V. SUMMARY In this paper, furniture fabric detection, global and local segmentation techniques were used successfully to segment a furniture type. The results obtained are fairly accurate in terms of fabric orientation and looks. Improvements need to be made in detecting contour more accurately. More

heuristic and intelligent method needs to be adopted to counter noise from fabric and background, and to process more complex furniture types.

ACKNOWLEDGMENT I would like to thank Prof. Bernd Girod for conducting a wonderful class on Digital Image Processing. Most of the techniques discussed in class were applied in this project. I would like to thank Andre Filgueiras for his excellent directions on using MSER for image segmentation and pointing out transform functions in Matlab. I would also like to thank Derek Pang and David Chen on their initial feedback on the project proposal.

REFERENCES

[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, Jitendra Malik, “Contour Detection and Hierarchical image Segmentation”, EECS Dept. Berkeley, CA, Feb 2010.

[2] D. Martin, C. Fowlkes, and J. Malik, “Learning to detect natural image boundaries using local brightness, color and texture cues,” PAMI, 2004.

[3] R. Unnikrishnan, C. Pantofaru, and M. Hebert, “Toward objective evaluation of image segmentation algorithms,” PAMI, 2007.

[4] Gary Bradski & Adarian Kaehler, Learning OpenCV )’Reilly publication, Kindle version.

[5] http://www.vlfeat.org/ [6] Anil K. Jain, Farshid Farrokhnia, “Unsupervised texture Segmentation

Using gabor Filters”, IEEE trans, SMC 1990. [7] Bovik, A.C., Clark, M., and Geisler, W.S. (1990), “Multichannel

Texture Analysis Using Localized Spatial Filters,” IEEE Trans. PAMI, vol. 12, no. 1, pp. 55-73.

VI. APPENDIX

Figure 5: Original Image

Figure 6: Gray image with Centroid locations and segmented image

Figure 7: Augmented image (with carpet like texture)

Figure 8: Augmented image without and with scaled luminance. Also, texture projection is most visible on the seat region.