LIU Xun(劉 栒), SHI Youqun(史有群), LUO Xin(羅 辛), ZHU Guoxue(朱國(guó)學(xué))

1 College of Computer Science and Technology, Donghua University, Shanghai 201620, China2 China Textile Enterprise Association, Beijing 100020, China

Abstract: Aiming at the lack of professional knowledge to guide apparel recommendation, an apparel recommendation method based on image design expert knowledge has been proposed. Then, apparel recommendation knowledge graphs have been created and a apparel recommendation question and answer(Q&A) system has been designed and implemented. The question templates in the apparel recommendation domain were defined, the task of recognizing the named entities of question sentences was completed by the Bi-directional encoder representations from transformer-Bi-directional long short-term memory-conditional random field(BERT-BiLSTM-CRF) model, and the question template with the highest matching degree to the user’s question was obtained by using term frequency-inverse document frequency(TF-IDF) algorithm. The corresponding cypher graph database query statement was generated to retrieve the knowledge graph for answers, and iFLYTEK’s voice application programming interface(API) was called to implement the Q&A. The experimental results have shown that the Q&A system has a high accuracy rate and application value in the field of apparel recommendations.

Key words: expert knowledge; apparel recommendation; knowledge graph; question and answer(Q & A) system; speech recognition

Introduction

In recent years, with the rapid development of clothing e-commerce, online shopping clothing has become a popular trend. However, the massive growth in the amount of online clothing information has made it difficult for users to shop for their preferred and suitable clothing. As a result, there has been an increasing amount of interest in the study of apparel recommendations. At present, most recommendation systems use the recommendation method based on the user’s purchase history. For example, apparel recommendations by the user’s history of visits, combined with visual features of clothing images, which requires the user’s history of visits as a prerequisite. Zhang[1]proposed a big data causal recommendation method that used user’s history and Bayesian network for causal discovery, and constructed an apparel recommendation model that considered only clothing brands and hues. At present, the most widely used recommendation method based on history is collaborative filtering algorithm, but it suffered from data sparsity and cold start problems that still need to be addressed. However, users rarely buy the same clothes when purchasing clothing[2]. Therefore, the traditional apparel recommendation based on user’s history[3]does not consider the user’s feature information. The results of apparel recommendations obtained by this method are somewhat blind. This recommendation method not only fails to personalize the recommendation, but also increases the cost of wearing. Compared with the recommendation method based on history, the recommendation method based on knowledge has higher application value in the field of apparel recommendations. Liuetal.[4]established a knowledge base by collecting expert opinions and literatures on clothing, expressed the knowledge by generative rules, and realized intelligent recommendation of clothing based on forward linking reasoning techniques. Feng[5]established a clothing knowledge system database based on the apparel recommendation experience provided by experts, and developed a personalized apparel recommendation system based on expert knowledge. Some studies[6-7]only focused on a single characteristic of the user, the scope of recommendation was limited, and the recommendation results were not practical.

Question and answer(Q&A) system[8]is an efficient form of natural language processing that can better recognize the intention of a user’s question than a search engine, leading to an answer that meets the user’s needs. Traditional Q&A systems rely on unstructured data crawled from the web and suffered from large amounts of redundant information and inefficient retrieval. The knowledge graph[9]as a semantic network, contains rich semantic information and provides a high-quality structured data source for Q&A systems, improving the accuracy and retrieval efficiency of Q&A. In recent years, Q&A systems based on knowledge graphs[10-11]have received widespread attention. For example, Duetal.[12]designed an intelligent Q&A system for Chinese knowledge graphs in the field of e-commerce, and realized knowledge Q&A in the field of e-commerce. Chenetal.[13]used knowledge graph technology to realize Q&A in the field of Chinese medicine. Current research has shown that the application of knowledge graph technology has improved the retrieval efficiency and accuracy of Q&A systems, but how to identify user’s intention and transform user’s question statements into query statements is still a problem that needs to be solved in order to build a Q&A system based on knowledge graph.

To address the above problems, this paper proposed an apparel recommendation method based on image design expert knowledge, and used knowledge graph technology to build an apparel recommendation intelligent Q&A system. First, the image design expert rule system was designed and the apparel recommendation knowledge graph was created. Then, question statement processing was performed, and question statement entity recognition was performed by the Bi-directional encoder representations from transformer-Bi-directional long short-term memory-conditional random field(BERT-BiLSTM-CRF) model, combined with term frequency-inverse document frequency(TF-IDF) text similarity algorithm to match question statement templates. Finally, cypher query statements were generated to retrieve the knowledge map for answers, combining with voice Q&A technology to enable users to quickly select the most suitable clothing for themselves.

1 Knowledge Graph Construction for Apparel Recommendations

The key to the expert knowledge-based personalized apparel recommendation lies in the design of clothing domain rules and the construction of knowledge base. In this paper, the information association between human body features and clothing features was achieved through the knowledge graph of apparel recommendation, and the image design expert rule system and the construction method of the knowledge graph were introduced below respectively.

1.1 Image design expert rule system

This paper addressed the lack of professional knowledge to guide apparel recommendations, combined with the knowledge of image design experts to provide a comprehensive quantitative analysis of three elements of aesthetics(style, color， and fabric) and concluded a set of quantitative aesthetic criteria suitable to apply online. More than 50 labels were designed for a single garment, including garment color, category, material, pattern, style, and occasion labels, as shown in Table 1. At the same time, combining expert matching experience and relevant information, the user image was quantified into a label system, which contains seven skin colors, nine face shapes, ten body types characteristics and height-to-weight ratio, as shown in Table 2.

Table 1 Clothing label information

Table 2 User image information

Based on the above-mentioned labeling system, we proposed a personalized apparel recommendation method based on the knowledge of image design experts, combined with the apparel recommendation experience of image design experts, and summarized the matching rules of user characteristics and clothing characteristics elements. Firstly, we considered the matching rules of the elements of clothing characteristics themselves and used fuzzy logic to express the degree of matching of clothing, where 0-0.1 means “very poorly matched”, 0.2-0.4 means “poorly matched”, 0.6-0.8 means “matched” and 0.9-1.0 means “very well matched”. The matching rules for some of clothing categories are shown in Table 3, with jeans and shirts having a matching degree of 0.9, which means “very well matched”.

Table 3 Examples of matching rules for clothing categories

The apparel recommendations aim to provide users with professional dressing guidance, so it is important to consider not only the matching of clothing elements, but also the matching of clothing elements with elements of human characteristics. Body characteristics include skin color, face shape, body shape and height-to-weight ratio, which are mainly considered to match with clothing style and color rule. Face shape, body shape and height-to-weight ratio determine the match between the user and the clothing style, and skin color and height-to-weight ratio determine the match between the user and the clothing color. Specifically, the choice of collar shape has a direct impact on face shape, and the right collar shape can flatter the face. Both body type and height-to-weight ratio correspond to the overall label of the clothing style. Body type and height-to-weight ratio directly influence the choice of clothing style, and choosing the right clothing style can conceal and modify body flaws, amplify strengths. There is also a close relationship between clothing color and body weight, and the right color combination can visually correct the shape. According to the height-to-weight ratio of people, people have been divided into five types, and experts have given rules for matching these five types with clothing colors. For example, short thin people are suitable for mild colors in clothing with the same color scheme, and not for very dark colors or very light colors.

1.2 Knowledge graph construction method

Based on the image design expert rule system, a knowledge graph of apparel recommendation was constructed. The image design expert rules were stored in the Neo4j graph database in the form of triples of(clothing features, suit_for, body features), and the nodes in the knowledge graph were the label information of clothing elements and body feature elements. The relationship “suit_for” between clothing features and body features was defined, and expert rules were added to the relationships as weights in order to quantify and compare the relationships. The personalized apparel recommendations were achieved by using the semantic information contained in the knowledge graph. Firstly, the image expert rules were represented in comma-separated values(csv) form to construct the body feature entity body.csv, the clothing information entity clothes.csv and the relationship suit_for.csv. The classification labels corresponding to color, style, material, pattern and style were added to the clothing node as clothing feature attributes. Then classification labels corresponded to skin color, face shape, body type and height-to-weight ratio were tags as body feature attributes to the body feature node. Then, the csv files were imported via the “neo4j-admin import” command to generate the nodes and relationships. Finally, the apparel recommendation knowledge graph was successfully constructed.

2 Q&A System Design

Current recommendation systems have been well used in various areas, but lack efficient and intelligent human-computer interaction in the field of apparel recommendation. In order to address this shortcoming, this paper has adopted the form of voice Q&A, combined with the apparel recommendation knowledge graph, to make accurate and efficient apparel recommendation solutions for users according to expert rules. The Q&A system in this paper has been divided into a question semantic understanding module and a knowledge base answering module, and the details of the two modules are as follows.

2.1 Question semantic understanding module

After being obtained the question text by calling the speech recognition application programming interface(API), the function of question semantic understanding was implemented mainly by entity recognition and template matching methods.

2.1.1Appareldomainentityrecognition

To address the complex entity naming rules in the apparel domain and the incorrect recognition of entity boundaries by traditional entity recognition models, this paper has annotated the apparel recommendation corpus with begin-inside-outside(BIO) in a semi-automated manner and used it as input to the BERT-BiLSTM-CRF entity extraction model. The entity recognition model BERT-BiLSTM-CRF in this paper not only resolved the problem of poor recognition of ambiguous words by the word to vector(word2vec)[14]model, but also could well fuse the contextual information of words and contextual label information. First, word vectors of interrogative sentences were obtained by training the BERT model[15]. Then, the BiLSTM[16]was used for context encoding to obtain feature information for named entity recognition. Finally, the output sequence of the BiLSTM was annotated by the CRF[17]layer to obtain the global optimal sequence. The framework of the entity recognition model was shown in Fig. 1, which used a corpus of BIO annotated clothing questions and the answers as the model training dataset with five labels, namely “O”, “B-Body”, “I-Body”, “B-Clothes”, and “I-Clothes”. The model training was implemented in Python 3.7 and the training environment was built on the Tensorflow platform. The training environment was based on the Tensorflow 2.2.0, Ubuntu 16.04 operating system, and Tesla K80 GPU accelerated graphics card. To prevent overfitting problems, the dropout was used in the input and output of the BiLSTM with a value of 0.5.

Fig. 1 Frame of BERT-BiLSTM-CRF model

The Q&A system in this paper was based on the BERT-BiLSTM-CRF entity extraction model, which solved the problem that the traditional word vector representation was single and could not deal well with the multi-sense characteristics of words. Even for semantically complex question sentences, the apparel recommendation-related entities could be extracted from the questions, and combined with question template matching to complete question recognition.

2.1.2Questiontemplatematching

There are three main approaches to question and sentence parsing, including template matching-based approaches[18], semantic extraction based on syntactic analysis[19], and semantic extraction based on short text similarity[20]. The apparel recommendation domain is a proprietary domain with a limited variety and scope of questions, so this paper has used template matching for question and sentence parsing. And user’s intention matching is performed by calculating the similarity between the questions and template questions. The specific flow of template matching in this paper is as follows.

According to the apparel recommendation rules, this paper designed six types of question templates based on user’s intention, and the information of the set of question templates were shown in Table 4. Among them, user’s intention was further divided into objective intention and subjective intention. Objective intention refers to entities, attributes and relationships that can correspond to the knowledge graph; subjective intention cannot be mapped to specific entities and does not require retrieval of answers in the database. For subjective intention, the corresponding response templates were set and output directly to the user through speech; for objective intention, the corresponding clothing entities or human characteristic entities need to be identified through entities. The question templates with the highest similarity were obtained through similarity matching with the question templates, and then the questions were transformed into cypher query statements, and the corresponding relationships or attributes were retrieved in Neo4j, and these entities and the corresponding relationships were filled into the response template to generate natural language answer sentences.

Table 4 Problem template set corresponding to different intentions

Based on the entities identified in the entity recognition session, text similarity matching was performed with the question templates, and the question template with the highest similarity was selected. In this paper, the TF-IDF algorithm[21]was adopted for text similarity matching.

(1)

whereTdenotes the TF of the word in the problem. The largerTis, the more representative it is in the problem.Adenotes the number of occurrences of a word in the question, andSdenotes the total number of words in the question.

The IDF value was calculated by

(2)

whereIdenotes the IDF value. The product ofTandIrepresented the weight of the word in the current problem.Ddenotes the total number of template questions, andD′ denotes the number of template questions containing the word.

A weighted sum of all the word vectors in the problem was performed to obtain the template problem vectorXand the user problem vectorY. The cosine between the sentence vectors was calculated by

(3)

whereCwas denoted to obtain the similarity between the proposed problem and the template problem.

To ensure the accuracy of the questions and avoid meaningless searches of the knowledge base, a threshold of 0.5 has been set for similarity, and when the calculated similarity is greater than the threshold, a reply or query database operation would be performed. Otherwise, the user would be prompted to re-enter the question.

After the entities in question have been identified and matched to the template questions, the relationship names have been obtained based on the mapping relationship between the template questions corresponding to the user’s intension and the database, and combined with the names of the identified apparel recommendation entities, cypher query statements have been generated according to the rules. The corresponding relationship between question sentences and query sentences is shown in Table 5.

Table 5 Examples of cypher query corresponding to user’s questions

The cypher query statement retrieves answer entities or attributes queried from the Neo4j graph database, filled in with answer sentence templates, and generated natural language answers, finally the answer sentences were output via speech.

2.2 Knowledge base answering module

2.2.1Recommendationmethod

The apparel recommendation method in this paper was by mapping human body characteristic elements to clothing characteristic elements, as shown in Fig. 2.

Fig. 2 Mapping relationship between the two types of characteristic elements

The suitabilitySi(i=1, 2, 3, 4) in the graph is the weight of “suit_for” relationship between human body features and clothing features, andS′i(i=5, 6) indicates the suitability of the clothing itself. The user inputs the information of human body features, gets the suitability of the corresponding clothing features from the apparel recommendation knowledge map, and calculates the comprehensive suitability score of the recommended clothing according to

S=W1S1+W2S2+W3S3+W4S4+S′5+S′6，

(4)

whereW1+W2+W3+W4=1,Wi(i=1, 2, 3, 4) indicates the influence score of each human body feature in apparel recommendation, which is given by experts, and the corresponding influence differs for different user features. In the apparel recommendation process, the user pattern that matches all the information is first found. Then the matching degree of clothing and user is calculated and arranged in descending order. Finally, the clothing of the top six clothing patterns are recommended for the user.

2.2.2Answersentenceconstruction

In this paper, a template approach was used to generate answer sentences and six corresponding answer sentence templates were designed based on six different types of user’s intention, as shown in Table 6.

Table 6 Answer template corresponding to user’s intention

By recognising the user’s intention and matching the appropriate reply sentence template, the reply sentences would eventually be output in natural language by the speech synthesis function of the speech technology module. For example, if the user accepted the recommended result, the corresponding “thank you” sentence would be replied to “Glad to help you, thanks for using it”. Another example is if the user types “What are the characteristics of a straight body type?” then we will identify the entity “straight body type” by the entity recognition model, determine the user’s intention to query the characteristics of the human body based on the similarity calculation, match the question template with the highest score, retrieve Neo4j with the cypher statement corresponding to the template, and get the attribute description information for “straight body type”. The attribute description information for “straight body type” is populated into the response template “The characteristics of {entity} are: {attribute}”, and the answer obtained is “The characteristics of straight body are: similar waist width, chest width, and hip width, with no curve in front”. If the user rejects this recommendation result, the next level of clothing will be recommended based on the relevance calculated by the recommendation method.

3 Experiments and Result Analysis

3.1 Speech recognition processing

This paper is based on iFLYTEK’s voice API implementation. First, we register the voice dictation application on iFLYTEK’s official website to get the corresponding APIkey and APISecret. Then, the system sends a Websocket handshake request to iFLYTEK’s server through the authentication mechanism, and uploads and receives data through the WebSocket connection after a successful handshake. Finally, the returned JavaScript object notation(JSON) data are parsed as the result of speech recognition. The recognition process as shown in Fig. 3.

Fig. 3 Speech recognition flow chart

The formats of the audio are sampled according to the standards shown in Table 7.

Table 7 Audio sampling standard

According to the string parsed from the JSON returned by API, the text of the question is extracted and thus further processed.

3.2 Analysis of expert knowledge recommendation result

3.2.1Analysisoftheresultsoftheexpertknowledge-basedrecommendationmethod

The dataset for this article comes from real questions related to apparel recommendations crawled from the Q&A website about Chinese clothing. We have got 672 real Q&A statements related to apparel recommendations. The recommendations are tagged based on the answers adopted by users in the website, and the content of the adopted answers is tagged as the correct recommendation result. The crawled Q&A statements were filtered for six categories of user intention, and finally 50 Q&A statements corresponding to each category of user intention were obtained, making a total of 300 apparel recommendation-related Q&A statements. The Q&A dataset was then divided into a training set and a test set in a ratio of 8∶2.

In this paper, a apparel recommendation knowledge graph was constructed and a recommendation method based on expert knowledge was proposed. To verify the effectiveness of this paper’s recommendation method, we used precision and recall as evaluation metrics to compare it with traditional collaborative filtering algorithms. Recall rate is the ratio of the system’s recommendation results to the right recommendation results label. The greater the recall rate is, the more accurate the recommendation are. Accuracy rate is the ratio of recommendations accepted by the user to the overall set of recommendations. The higher the accuracy rate is, the better the performance of the recommendation system is.

In the recommendation process, withMas the set of recommended results from test users,Nas the set of items actually selected by users in the test set, the accuracy ratePand recall rateRwere calculated as

(5)

(6)

Five sets of experimental data were randomly selected to compare the precision of the expert knowledge-based apparel recommendation method with that of the collaborative filtering-based recommendation method as shown in Fig. 4, whereP1 andR1 denote the accuracy and recall rates of the expert knowledge-based recommendation method, respectively;P2 andR2 denote the accuracy and recall rates of the collaborative filtering algorithms, respectively.

Fig. 4 Comparison of the results of the two recommended methods

As shown in Fig. 4, the knowledge-based apparel recommendation method outperforms the collaborative filtering algorithm in terms of accuracy and recall rates, and has better apparel recommendation results.

In order to further verify the practicality of this paper’s method for apparel recommendation, apparel recommendations were made for three users with different characteristics, and the matching degree of clothing with the user’s body type was calculated according to Eq.(4), as shown in Table 8.

Table 8 Apparel recommendation results

As an example, the recommended result for No.1 is 0.8 for light skin tone and light green color, 0.9 for oval face and round neck, and 0.6 for tops with a higher influence of face shape. Thus, it has been calculated that the matching degree of user-1’s characteristic information and light green round neck T-shirt is 0.86, which can be considered as a better match, and this result was consistent with the rules of clothing matching and in line with the aesthetics of the public.

3.2.2Analysisoftheresultsoftheidentificationofquestionentity

As there is no public question and answer corpus dataset related to apparel recommendation, the training and test sets are divided in a ratio of 8∶2 based on the relevant data of the questioned sentences obtained from 3.2.1 of this paper. Then, BIO annotation was performed to construct an apparel recommendation domain corpus, and the annotated corpus was used to train a BERT-BiLSTM-CRF entity recognition model.

The entity recognition results showed that the BERT-BiLSTM-CRF model performed better in the entity recognition in the apparel recommendation domain. The effectiveness of the model has been assessed by the accuracy rateP, recall rateRand comprehensive evaluation indexF1, calculated as

(7)

(8)

(9)

whereTdenotes the number of correctly identified entities,F1denotes the number of all identified entities, andNdenotes the number of all tagged entities.

A comparison of the three named entity recognition models, BiLSTM, BiLSTM-CRF and BERT-BiLSTM-CRF, is shown in Table 9.

Table 9 Comparison of three models

In processing the apparel recommendation corpus, the BERT-BiLSTM-CRF model is performed best among the three models. The Q&A system in this paper was based on the BERT-BiLSTM-CRF entity extraction model, the model solved the problem that the traditional word vector representation was single and could not handle the multi-sense features of words well. For the interrogative corpus data of apparel recommendations, the BERT model was added to the BiLSTM-CRF model for fine-tuning, and the contextual features of the text were identified by a bi-directional encoder, which effectively avoided the accumulation of errors in word separation and improved the accuracy of the subsequent entity extraction work.

3.3 System interface display

The system has been developed by using the Flask framework and the Q&A interface is designed as shown in Fig. 5. The left sidebar implements the navigation of the page, and the content area in the middle enables graphical representation of the quiz results, which can be entered by text or voice, followed by the voice output of the results. The content area on the right is a detailed description of the entity nodes.

Fig. 5 Q&A interface display

4 Conclusions

In this paper, firstly, an intelligent apparel recommendation method based on image design expert knowledge is proposed. Then, the apparel recommendation knowledge graph is constructed as the knowledge base of the Q&A system by sorting out the expert rules. In addition, the entity recognition task of the Q&A system is done by the BERT-BiLSTM-CRF model. Finally, the system achieves personalized apparel recommendation with the help of speech technology. The results showed that the recommended clothes by the recommendation system in this paper were in line with popular aesthetics, and the human-computer interaction of voice Q&A improved the user’s retrieval efficiency.