matrix. J. Am. Chem. Soc, 128, 39, 12693– 12699, 2006.
15. Adleman, L., Molecular computation of solutions to combinatorial problems. Science, 266, 1021–1024, 1994.
16. Winfree, E., Liu, F., Wenzler, L.A., Seeman, N.C., Design and self-assembly of two-dimensional DNA crystals. Nature, 394, 539–544, 1998.
17. Benenson, Y., Paz-Elizur, T., Adar, R., Keinan, E., Livneh, Z., Shapiro, E., Programmable and autonomous computing machine made of biomolecules. Nature, 414, 430–434, 2001.
18. Ray, K.S. and Mondal, M., Classification of SODAR data by DNA computing. New Math. Nat. Comput., 7, 3, 413–432, 2011.
19. Ray, K.S. and Mondal, M., Logical inference by DNA strand algebra. New Math. Nat. Comput., 12, 1, 29–44, 2016.
1 *Corresponding author: [email protected]
3
Intelligent Garment Detection Using Deep Learning
Aniruddha Srinivas Joshi*, Savyasachi Gupta, Goutham Kanahasabai and Earnest Paul Ijjina†
Department of Computer Science and Engineering, National Institute of Technology, Warangal, India
Abstract
Garment detection is a complex image processing task that has a multitude of applications in the industry such as retrieval of similar garments, Artificial Intelligence–powered fashion recommendation models, and automatic labeling of catalogs. Retailers of fashion stores can benefit from knowing vital information about the types of garments that customers are interested in and thus ensure a more profitable business model. In this chapter, a novel framework is proposed for the detection of garments of interest from the footage of a surveillance camera. The video frames are processed using the GMG background subtraction model to obtain relevant foreground information along with foreground masks. The Mask R-CNN object detection model is used to identify customers and multiple other image processing techniques are used to obtain the active garments in these frames. The detected customers are tracked and the OpenPose human pose estimation framework is utilized on them to obtain useful landmarks. The garments of interest are then determined based on the filtration of confidence scores calculated for each active garment. The framework was tested on a CCTV video dataset and was found to be effective despite facing arduous obstacles such as background noise and occlusions.
Keywords: Garment detection, pose estimation, object detection, customer analytics, deep learning, computer vision
3.1 Introduction
Recognition of garments is a complex image processing task, which benefits a wide array of applications such as customer behavior analyses, forecasting sales, market segmentation [1], and computer-aided designs for fashion [2–4]. This task has attracted a lot of research interest in the field of image processing [5–8]. Identifying the garments that customers of a store are interested in can further aid the development of the aforementioned use cases.
As a retailer, understanding the needs of the customers and anticipating the future needs of the customer base can be beneficial in aiding to stock up on appealing products. This can prove vital to a growing business strategy in today’s ever so competitive world. The illustrated prospects in this research area motivated us to propose an approach that detects garments from surveillance videos and also indicates the extent to which a person is interested in a particular garment. We believe that the usefulness of analyzing customer behavior using machine learning shows promise for a wide range of applications, especially in those where stakeholders can benefit from these analyses. It is noteworthy that the task of garment detection has attracted research attention in the field of image processing, however, developing a system to recognize garments that appeal to customers from surveillance videos is a herculean challenge primarily due to several reasons as explained further.
Firstly, there are numerous sub-tasks involved which include customer detection, tracking, and clothing segmentation. Secondly, the identification of complex garments from indoor surveillance footage is complicated as these garments usually comprise of different textures, types of fabric, and a multitude of colors in addition to their deformable nature. Finally, as CCTV cameras are conveniently installed in stores at angles that make them suitable to monitor human behavior, many a time these customers themselves behave as occlusions by blocking regions of garments to be detected, thereby hindering the task of identifying the garments themselves, further escalating the complications.
In this chapter, we propose a novel framework for the task of identifying garments that appeal to customers and these garments are referred to as garments of interest. After the identification of garments in a video frame using background subtraction, we employ the Mask R-CNN object detection model [9] for the task of identifying customers in the store. We use the OpenPose pose estimation framework [10] to obtain the feature points on the human body, enabling us to draw a correlation between a customer’s wrist coordinates and a garment that the customer most recently interacted with. This allows us to derive a confidence score metric between a customer and the garment into consideration.
The chapter is organized as follows: Section 3.2 highlights the related works, Section 3.3 elucidates the proposed approach, Section 3.4 highlights the results obtained by the proposed approach, Section 3.5 highlights the major findings of this study, and Section 3.6 delineates the conclusion and scope for future work in this field.
3.2 Literature
Bu et al. [11] proposed a Multi-Depth Dilated Network (MDDNet) for the identification of landmarks on fashion items. Since garments and fashion items are often occluded in the environment of detection, the authors proposed an approach to identify the fashion landmarks by the introduction of a Multi-Depth Dilated (MDD) block. These MDD blocks are composed of a different number of dilated convolutions in parallel and are utilized in the MDDNet. A Batch-level Online Hard Keypoint Mining (B-OHKM) method is also proposed to extract hard-to-identify fashion landmarks during the training stage of the network, thus enabling the network to be trained in a manner that improves the performance of such landmarks. Although this approach achieves state-of-the-art performance on fashion dataset benchmarks, it is only effective in identifying generic clothing items such as shirts, pants, and skirts and cannot guarantee good results on complex garments with different textures and color overlays.
Yu et al. [12] proposed a model to identify fashion landmarks by enforcing structural layout relationships among landmarks that utilize multiple stacked Layout Graph Reasoning (LGR) layers. The authors define a graph called layout graph, which is a hierarchical structure with a root node, body-part nodes (eg. upper body, lower body), coarse clothes-part (eg. sleeves) nodes, and leaf nodes. Each LGR layer maps features into these structural graph nodes, performs reasoning over them using a LGR module, and then maps the graph nodes back to the features to enhance their representation. The reasoning module uses a graph clustering operation to get the representations of the intermediate nodes and performs a graph deconvolution operation over the entire graph. After stacking multiple such LGR layers in a convolutional network, a 1×1
