マルチメディアの分野で人物の画像を検索、分析、比較する技術「人物再認識」について研究されている王先生(価値交換工学社会連携研究部門 特任助教)。さらにその技術をファッション価値や人間行動の分析へ活用することにも取り組まれています。「人物再認識」によって可能になる未来、興味を持ったきっかけについてお伺いしました。

Dr. Wang is a specially appointed assistant professor in RIISE, where his research focuses on “person re-identification” – a technology for searching, analyzing, and comparing images of people. He is also working on applying this technology to fashion value analysis and understanding human behavior. We asked him about the future made possible by “Person re-identification” technology, and how he got interested in it.

-What is your research field?

Wang: My research field is multimedia content analysis and retrieval.

-Could you explain a little bit about your research theme, “person re-identification”?

Wang: Person re-identification is a technique for retrieving, analyzing and comparing person images across cameras, which can effectively help investigators discover and track targets in a large amount of video. It is one of the most challenging key problems in the field of multimedia content retrieval and social security at present, and a supporting technology for the discipline of big data behavioural analysis, which is very important for improving the efficiency of video search, the detection rate of criminal cases, and tracking virus carriers.

Figure 1: The example of person re-identification, which is a technique for retrieving, analyzing and comparing person images across cameras. 

-You are also conducting research on fashion value analysis and understanding human behavior as an application of person re-identification. How did you become interested in this topic?

Wang:  My research interests include multimedia content analysis and retrieval. I focused on person re-identification when I was a PhD student, since person re-identification is a valuable and hot topic in China. When I moved to Japan as a postdoc, I had some fundings from JSPS and Microsoft. Both fundings are related to person re-identification. That is why most of my publications are in this field.  I started to investigate the topics of human behavior and fashion value, when I joined RIISE institute, The University of Tokyo. Because I do want to have some change to myself, and I am still trying to find the topics fitting Value Exchange Engineering’s requirements.

-What is fashion value analysis?

Wang: Each day, we collectively upload to social media platforms billions of photographs that capture a wide range of human life and activities around the world. At the same time, computer vision technologies, such as semantic segmentation and visual search, are seeing rapid advances and are being deployed at scale. With large-scale recognition available as our fundamental tools, it is now possible to ask questions about how people dress, buy and sell clothes across the world and over time. In particular, I ask: can we detect and predict the fashion value of one focused cloth socially and individually? The purpose of the topic fashion value analysis is to assign a new value to a focused cloth/shoes/bag. In my opinion, the value of a product consists of three parts: 1) Basic Value, which is decided by the product’s content, quality, materials, brand, and local price level. 2) Social Fashion Value, which can be influenced by the social environment. For example, when a super idol wears a pair of shoes with a special pattern, the pattern might be popular and the shoes would have more value. Here is another example, when a lot of office ladies in Roppongi take a new style of “Samantha Thavasa” bag, this bag might be fashionable and arouse more interest. 3)  Individual Fashion Value, which depends on the requirements of each person. If a person already has a pair of blue trousers, according to a suitable collocation of clothes, one pair of brown board shoes would have more value for him. To this end, a product will be re-valued by additional fashion value. This will benefit commerce and balance the requirements between the clients and merchants.

-Your sub-topics are Fashion Retrieval and Cloth Retrieval, Could you expand on these topics a little bit?

Wang: Fashion retrieval is a challenging task, which requires searching for exact items accurately from massive collections of fashion products based on a query image. It still has limitations for application to real-world visual searches. The main reason for this is the common nature of fashion images captured under uncontrolled circumstances (e.g. varying viewpoints and lighting conditions). In particular, fashion images are vulnerable to shape deformations and suffer from inconsistency between the user’s query images and refined product images [1]. Considering the similar challenges, researchers start to use person re-identification models to conduct fashion retrieval [2]. 

I also had some preliminary experiments. On the DeepFashion2 [3] dataset, I got 73% accuracy at Top@10 with the method [4] and 75% accuracy at Top@10 with the FasetReID method [5]. It is far away from the SOTA methods. 

Figure 2: The example of fashion retrieval [1]. 
[1] Sanghyuk Park, et al., Study on Fashion Image Retrieval Methods for Efficient Fashion Visual Search, CVPR Workshop, 2019
[2] https://synerise.com/papers/fashion-retrieval-with-person-re-identification-models
[3] DeepFashion: http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion.html
[4] Part-Aligned Bilinear Representations for Person Re-identification, ECCV, 2018
[5] FastReID: A Pytorch Toolbox for Real-world Person Re-identification. 

-How do these studies relate to understanding human behavior?

Wang: Human behavior is driven, in part, by thoughts and feelings, which provide insight into the individual psyche, revealing such things as attitudes and values. Human behavior is shaped by psychological traits, as personality types vary from person to person, producing different actions and behavior. Extraverted people, for instance, are more likely than introverted people to participate in social activities like parties. Human behavior understanding is a very huge topic. We start to research human behavior from the perspective of important person localization [pub1] and human location prediction [pub2].

[pub1]  Very Important Person Localization in Unconstrained Conditions: A New Benchmark, AAAI, 2021
[pub2] Location Predicts You: Location Prediction via Bi-direction Speculation and Dual-level Association, IJCAI, 2021

–Do you consider fashion retrieval and human behavior understanding two separate topics? 

Wang: Although they are separated topics, both of them are considered as the basis of the research of fashion value. Figure 3 shows the framework of my fashion value system. It includes four levels from bottom to top: 1) The first level is my research background. Related technologies can be taken as a support for fashion information extraction. 2) The second level is low-level value analysis, consisting of cloth parsing, attribute learning, photo based retrieval, sketch based retrieval and text based retrieval. In this level, the system will extract useful fashion information from personal photos that are captured in the street, recorded in social media websites such as Facebook, Flickr, and Instagram, and searched log in e-commercial shops. 3) The third level is high-level value analysis, where we exploit big data analysis methods on the platforms of social media, GIS media and our life log, and obtain social and personal fashion value of all related products, in terms of space, time, crowd, neighborhood, and individual.  4) The fourth level is value generation, based on the low- and high-level fashion value analysis. 

My research topic “person re-identification” is the key technology for fashion and cloth retrieval. Hence, the topic “Fashion Retrieval” is in the second level of the framework, i.e., low-level value analysis. On the other hand, the topic “Human behavior understanding” is in the third level of the framework, i.e., high-level value analysis.

Figure 3: The framework of my fashion value system. 

-Did you get the idea for your research from looking at the Mercari service?

Wang: I discussed with Professor Yamasaki., who had previously done research on fashion retrieval.Fashion retrieval is a task highly related to person re-identification. We predicted  Mercari may have some needs on cloth retrieval. So I made the fashion value idea.

-How is the progress of your current research?

Wang: Currently, I had some preliminary results on fashion retrieval based on my previous research background. I also had two papers accepted in top conferences that are related to human behavior understanding.