The analysis shows that most hotels (26 out of 35) responded to reviews within a week. According to the findings about response time to complaints [16], customers deem a response made within 7.9 days as acceptable. In this sense, the hotels in this study generally performed well in timeliness. Timely response should be expected by most (if not all) customers, especially those who raised complaints in the reviews. This expectation should be exceptionally high for luxury hotels as they are supposed to outperform the low-class counterparts in all service aspects. A delayed reaction or service recovery would threaten the reputation of service provider [23]. Complainants would consider timely response as something which a service provider should prioritize because their feeling of uncertainty and negative emotion need to be promptly addressed [24]. Therefore, it was even argued that additional compensation is not necessarily more important than timely response in addressing a complaint [25].

### 4.2 Response Style and Tone

Our perusal of management responses made by the seven chain hotels revealed that repetition of contents in customer review was salient in positive reviews but not in negative reviews. The hotels might not want to reinforce the incidence or service failure in their response to negative reviews. They generally acknowledged the problems and expressed apologies. In contrast, when responding to compliments in positive reviews, the hotels might want users to pay attention to their good deeds. It may also be due to the fact that not much can be written to address compliments. It is worth-noting that lengthy responses demonstrate hotel's serious attitudes toward consumer feedback and enhances customer satisfaction [6]. Therefore, repetition of the compliments is a viable approach. Furthermore, we found that repetition of contents is frequent if hotel staff is praised. This is important as hotel is a service industry where staff is the most valuable capital.

Regarding the contents in responses to positive review, responders generally included structural components suggested in previous studies which are greeting, expressing feelings, thanking reviewers, and continuing relationships, with some of them also recognizing the reviewer's value [9, 18, 26]. However, when facing customer complaints, their response style exhibited some variations. This approach is important as personalized responses help regain customers' trustworthiness [27]. The personalized messages signal the hotel's concerns about the customer's opinions and the high level of hotel engagement [28]. Therefore, in the case of service recovery, management response should not be standardized [29].

Response tone can generally be categorized into conversational tone and professional tone. A conversional tone refers to an engaging and natural style of organizational communication as perceived by consumers [30]. Conversely, professional tone means a relatively standard response which is more respectful, formal, and factsoriented [8]. An example of response consistent with professional tone is shown below:

"Thank you for taking your precious time to share your experience at Grand Hyatt Hong Kong.

We are most delighted to hear that you enjoyed your stay in a room with stunning view of the harbor, and the exceptional service provided by Osman, Grace, and our Guest Experience Team has enhanced the experience. We will definitely pass on your kind words of praise to the team to recognize their efforts. Your compliments on our scrumptious breakfast spread is also much appreciated.

Thank you for your perfect rating and we look forward to welcoming you back in the nearest future" . [ 31 ]

In this example, it can be observed that the writing is formal and contains respectful words such as "thank you" and "much appreciated". The response also emphasizes on the facts which the customer appreciated, for examples, "stunning view", "exceptional service", and "scrumptious breakfast spread".

The following shows an example of response featuring conversational tone:

"Firstly, thank you for staying at Kowloon Shangri-La, Hong Kong for having taken the time to share your feedback with us.

Please accept my sincerest apologies for the inconveniences that you have experienced regarding your room reservation. I have personally reviewed the whole situation as described with my Reservations Manager. I concurred with you that you should expect nothing but a good experience throughout the reservation process, especially being our loyal guests. Please rest assured that we are able to make necessary improvements to enhance our guest's booking experience.

Despite of this matter, l am very thrilled to know that we provided you with an enjoyable stay with us.

Indeed, it's always our pleasure to take care of you. Thank you very much for your compliments about the Hotel and our friendly services. We are looking forward to welcoming you back at the Hotel again real soon" [ 32 ].

In this example, the tone is relatively less formal, such as the phrases "firstly", "I am very thrilled", and "it's". The third and fourth paragraphs of this response sound natural as of s/he was talking to the guest. This communication style was recommended when customer reviews are negative because it is beneficial for organizational reputation [33]. The natural style of communication helps shorten the psychological distance between the hotel and guest, which will in turn increase the effectiveness of service recovery [34].

As this study also investigates whether chain hotels within the group had different response styles and tones, we observed if variations exist during our analysis of the data. We found that the tone used by hotels in Hyatt group and Regal group was consistent. They commonly used a professional tone in responding to positive or negative customer reviews. However, hotels in Shangri-la group adopted both professional and conversational tone approaches. The adopted approaches seem varying with the responders. We found that two managers were assigned to respond to online customer reviews. The resident manager preferred using a conversational tone in responding to all types of reviews. In contrast, the general manager preferred using a professional tone, especially in responding to positive and neutral reviews. The findings echo the argument that communication styles vary with the socio-demographic background of responders [35]. While general manager is at the executive level of the organization, a professional (formal) tone should be coherent with his/her status. Responder at the middle management level (e.g., resident manager) should be more suitable to use conversational tone.

Regarding response contents, hotels in the same group generally adopted similar structure components including greeting, expressing feelings, thanking reviewers, and continuing relationships, in responses to positive reviews. However, there were also exceptions. Regal Hong Kong Hotel preferred to use similar sentences to respond to every positive reviews. However, Regal Airport Hotel managers employed varied words to respond. On the other hand, more variations between chain hotels in the same group were observed in responses to negative reviews. Grand Hyatt Hong Kong preferred to explain the cause of the problem and was self-justifying in responding to problems that were not caused by the hotel. Conversely, Hyatt Regency Hong Kong Sha Tin would not mention problems in detail or offer explanations but instead directly apologizes for any inconvenience caused.

According to Ho [18], acknowledging problems helps maintain relationships with customers and is the most effective way to reduce customer dissatisfaction. Denying a problem should be avoided in the situation of service failure. Surprisingly, we found that these practices were not consistently pursued. Kowloon Shangri-la Hong Kong tended to respond to negative reviews by acknowledging the problems, whereas Island Shangri-la Hong Kong tended not to mention the problems stated in the customer review while neither acknowledged nor denied the problems. Additionally, they typically used similar content to respond to negative reviews. Kerry Hotel Hong Kong made relatively brief responses. Although it was willing to admit the problems, it seldom expressed apologies. The findings show that the chain hotels in the same group did not use a consistent approach.

### 5 Contributions and Managerial Implications

Online reviews and management responses are both important in influencing customer decision making and hotel performance, therefore hotels should devote effort to manage eWOM. In view of the high service standard of luxury hotels, useful insights can plausibly be drawn from their responses. While scholarly attention on luxury hotel management response was scarce, the current research fills the gap by unveiling the response characteristics, style, and tone of this hotel class, so that a comprehensive theoretical framework can be developed in the future.

Findings of this study show that most hotels responded to all review regardless of the review valence. This practice should be strictly followed by other luxury hotels as response hints that the hotels value and care their customers. In general, the responses were made within a week which can be a benchmark for luxury hotels in determining the performance indicators of the team responsible for managing customer reviews. As customers generally expect a prompt response, hotels should strive to respond even quicker but without sacrificing any accuracy. The findings show that responders' job positions were primarily at the management level. This industry norm signals that luxury hotels should not assign an operational staff to respond to the customers. Otherwise, customers might feel disrespectful.

Regarding the response style, professional and conversational tones were used. However, if the review was negative, a conversational tone was salient while message was personalized. These practices appear reasonable as a conversational tone helps shorten the psychological distance between the guest and hotel, while personalized message demonstrates hotel's sincerity in addressing the customers' problems. This approach is worthwhile for other luxury hotels to follow. On the other hand, the findings generally conclude that chain hotels in the same group adopted different response styles and tones. The appropriateness of inconsistency approach is controversial. Customers may expect consistent service from hotels in the same group. This expectation may be more salient for frequent customers and those who have joined the membership of the hotel group. The group may need to provide guidelines for its hotels in their response to online reviews.

### 6 Limitations and Future Research

This study has some limitations. First, while a qualitative approach was adopted, we have to ensure a manageable data size. So, only recent reviews of a small number of hotels were analyzed. Robustness of the findings was threatened. Future research is recommended to adopt a quantitative approach such as big data analytics. Given the findings of this exploratory study, hypotheses can be developed and validated in the future, for instance, the impact of a response on subsequent reviews and the impact of review topics (e.g., safety issues) on management responses. Second, to reduce any language bias, only English reviews and responses were analyzed. We do not know if language would make a difference on the findings. Third, this study only considered hotels in Hong Kong. Findings may be different if the case was from other places. Given these limitations, future research is recommended to enlarge the sample size, incorporate non-English reviews, and focus on hotels in other regions and countries. Comparisons can also be made between English and non-English reviews (and responses), and responses made by chain hotels located in different regions and countries, as well as responses made by chain hotels versus independent 5-star hotels. Succeeding studies can examine if and how the socio-demographic background of responder influences the responses, especially if the responses are personalized. Another interesting future study would be whether responders' names (Chinese versus non-Chinese) would affect user's evaluation of responses. Although the findings and implications were drawn from the practices of luxury hotels, it does not mean that their practices are the best. Future research should evaluate the responses using primary research approach, for example, survey, experiment, and even mixed-method approach so that broader and deeper insights can be drawn.

Acknowledgement. This project was partly supported by a research grant funded by the University of Macau (Ref.: CPG2021-00036-APAEM).

### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### The Usage of Emoji in Tourism-Related Instagram Posts: Suggestions from a Marketing Perspective

Viktoria Distel, Roman Egger(&) , Ugljesa Petrovic, Viet Linh Phan, and Simon Wiesinger

Salzburg University of Applied Sciences, Salzburg, Austria {vdistel.imte-m2019,roman.egger, upetrovic.imte-m2019,vphan.imte-m2019<EMAIL_ADDRESS>
Abstract. The relevance of emoji in social media marketing has attracted tremendous interest from academics and marketing professionals alike ever since emoji became a fixed component in user-to-user and business-to-user communication on online platforms such as Instagram and Facebook. Using a quantitative research approach in the form of a self-administered online survey in an experimental setting, the purpose of this study is to investigate what type of emoji positively impacts consumer behaviour, purchase intention, and user interaction in tourism-related Instagram posts. This research is novel in that it bridges the usage of emoji in the context of social media and tourism. The findings support tourism managers in the practical use of emoji for social media marketing campaigns on Instagram and show that (positive) face emoji evoke more positive emotions than non-face emoji.

Keywords: Emoji Instagram Social media advertising Tourism marketing

### 1 Introduction

With the rise of digital communication, social media platforms such as Facebook, Twitter, and Instagram have transformed into powerful communicational mediums as they allow users to express feelings, share everyday life moments, and connect with others [1]. In particular, Instagram, initially launched as a photo-sharing platform, has become one of the world's most popular channels, with over one billion active users. Beyond its visual-centred nature, Instagram acts as a network for several industries such as services, marketing, and tourism to grow their business. Increasingly, the potentials of Instagram have also been recognised by tourism service providers and destination marketing organisations (DMOs) for the creation of effective marketing strategies [2].

In addition to using videos and images for marketing purposes, emojis offer brands the opportunity to connect with the target market [3] on Instagram using an emotional touch. Essentially, emojis are facial expressions that allow users to express their feelings, moods, or emotions. As graphic symbols that show vehicles, food and beverages, weather, flags, animals, and so on [4], two main types of emoji can be identified; namely, face emoji (expressing human emotion and feelings, e.g., ) and non-face emoji (which do not involve human faces, e.g., ❤ ) [5]. Emoji can emphasise written text language as they are the most evolved form of available punctuation [6].

Seeing that emojis and emoticons are highly pervasive in our daily lives [7], recent studies have focused on customer purchase intentions relating to the appearance of emoji in advertising [8, 9]. For instance, scholars suggested that experimental purchases such as tourism products and services can stimulate emotions that positively increase emotional behaviour [10]. That is, travel-related products tend to influence emotions, whereby emoji can play a crucial role since they express an emotional mood as well. However, while emoji usage in tourism enterprise communications has been highlighted [11], the linkage between emoji and their significance in tourism advertising remains fragmented.

Meanwhile, although earlier literature has discussed Instagram advertising of DMOs [1, 12] and highlighted the vital role of user-generated content (UGC) on destination image formation [2, 13], the relation of user-generated content and the usage of emoji in tourism-related posts on Instagram is still an underexplored area. Likewise, while some scholars have looked into emoji and their meaning in regard to customer attention [14, 15], there are limited studies that focus on emoji and their impacts on the efficiency of tourism-related advertisements, especially on Instagram [16].

Owing to the rising interest in emoji around various research streams [16], this study aims to contribute to tourism research by examining what type of emoji attracts the most attention on tourism-related Instagram posts as well as what type of emoji creates more positive emotions in such advertisements. Additionally, this study assesses the type of emoji that creates the most user interaction. The specific research questions are as follows:

From a practical point of view, this study's findings can support destination management and tourism marketers in optimising their social media advertising and effective involvement of emoji in Instagram posts.

### 2 Literature Review and Hypothesis Development

### 2.1 Emoji and Social Media

As part of computer-mediated communication, emoji are widely used, and some emoji have nearly achieved universal understanding [17]. Emoji have been used in different ways and contexts on social media platforms including Instagram, Facebook, Twitter, and WhatsApp. They allow people with different cultural backgrounds to communicate more effectively and initiate interactions when necessary. It is believed that emoji can solve potential misunderstandings experienced in intercultural communication [18]. According to theory in social semiotics, emoji serves as a means of promoting shared values and affiliation between users [11]. This concept is expressed via a semiotic that establishes interpersonal meaning within the attitude system and a semiotic establishing experiential meaning.

When emoji were used, readers found a higher level of positive emotion than when no emoji were presented [17]. For non-face emoji, even though they do not show facial expressions, they still bring positive meanings to a message. Furthermore, emoji help to convey the meaning of a message more efficiently [19]; they can significantly reduce the ambiguity of a sentence and are able to change one's interpretation in terms of affection [5]. Hence, emoji can help to catch the attention of readers in a shorter period of time [15] and help to create a digital message with nuances and depth, resembling a face-to-face conversation more closely [14]. Other scholars suggested that a message is also considered to be more personalised when that message includes emoji [15]. They also point to the elaboration likelihood model, which explains the influence of emoji on the reader's attention. The model implies that information may persuade people with positive cues when they can be processed without much thinking, and personalisation can positively affect information processing [20].

Facial emoji encourage users to express different emotions that may not be as easily portrayed in online communication. People select suitable emoji that fits their current emotion/mood from a predefined list of emotional expressions. While, during the process of exchanging information on social network sites, this may seem like an extraordinary action, this is in fact an exercise that fosters our emotional honesty because, often times, individuals may be unable to express their true feelings due to social barriers [14]. Moreover, notably, visual images deliver higher emotional effects than just words [19]. Using emoji in a non-judgmental setting thus helps to understand our emotional state and to communicate it more openly [14]. However, this depends on the specific context and the individual perception of the respective emoji. Regardless, emoji convey a particular emotion and reinforce a message [21], which support us in conveying our socially challenging emotions and, by that, in acknowledging and accepting that emotion as a part of ourselves [12]. Accordingly, the following hypothesis has been formulated:

H1: The appearance of positive face emoji in tourism related Instagram posts raises more positive emotions than non-face emoji do.

### 2.2 Emoji, Consumer Behaviour, and Purchase Intention

While many functions and facets of emoji in communication have been explored in social media research, the usage of emoji and their effect on consumer behaviour and purchase intention is another essential component that needs to be considered when aiming to gain insights on the precise usage of emoji. To date, emoji are widely prevalent in marketing campaigns [9]; in other words, companies pursue new ways of communication that may positively influence the emotions of potential consumers. Hence, emoji are well recognised as a tool for social media advertising [3]. Recent literature suggest that emoji in promotional materials encourage users to respond to the message [23] and, regarding hedonic products, increase the effect of purchase intention [8]. Other scholars report an increasing number of emoji in marketing campaigns and point out that emoji in advertisements can positively impact targeted consumers because users appreciate the service more and the level of their satisfaction increases [9].

Notably, stimulation of consumers' emotional behaviour is particularly relevant when it comes to experience-orientated products, such as tourism services [10]. This study assumes the purchase intention of potential consumers to be equivalent to travel intention since an advertisement from an official destination management organisation on Instagram aims to bring travellers to the promoted destination. This has led to the following hypothesis:

H2: The appearance of emoji in tourism-related Instagram advertisements influences users' travel intention.

### 2.3 Instagram and Emoji in Tourism

Instagram has become a popular platform that triggers creativeness and self-promotion of tourism destinations [12, 22]. When applied strategically, user-generated content (UGC) can be highly beneficial for tourism organisations [24] because it enhances the destination image [12, 24]. Seeing that travellers can be viewed as low-cost marketers and brand ambassadors, research has found that pictures taken by users and reposted by DMOs improve engagement on Instagram [25]. Other scholars emphasise UGC as a crucial factor for destination image because user-generated Instagram pictures tend to present more diverse attractions than traditional brochures and travel guides [26]. As such, more tourism operators use Instagram to promote destinations and consider Instagram to be an online travel album where all tourists have access [12].

In terms of emoji usage in the tourism industry, a transformation in company communications including emoji on social media platforms has appeared [27]. As a result, travel organisations use emoji when communicating with tourists, leading to an increased human touch in business communications. Tourism is an industry that should strongly rely on the usage of emoji when considering tourism enterprises and their social media strategies as the integration of emoji brings more fun and friendly interactions to the customer base, which eventually leads to better business development [28]. In particular, it has been recognised that non-face emoji enhance reader trust by reducing message ambiguity [5]. As such, this study tests for the types of emoji that help to create effective UGC with the following hypothesis:

H3: The appearance of non-face emoji in tourism related Instagram posts helps to promote more engagement (likes, shares, and comments) than face emoji posts do.

### 3 Methodology

This research adopted a quantitative method approach with an online questionnaire in an experimental setting. More specifically, a within-subjects research design was applied (see Fig. 1) in which participants were asked to randomly evaluate a set of two pictures with a caption with text and a face emoji, two pictures with a caption with text and a non-face emoji, and two pictures with a caption with text only. The images were selected in a way that they do not show landmarks or people and mix indoor and outdoor shots.

The emoji were selected based on "The Most Frequent Emoji" list in 2019 [29], with the chosen ones being for face emoji and for non-face emoji. Under each Instagram post, a list of questions were asked to assess participants' attitudes, travel intention, and feeling (e.g., joy, interest, and contentment) towards the post. They were also asked to rate their perception of the picture and reactions to the post (i.e., like, share, and comment) based on a 6-point Likert scale to avoid the central tendency bias.

Participants were recruited via email and personal Instagram accounts using a nonprobability and purposive sampling technique. The inclusion criterion were adult participants over 18 years old as they are legally entitled to purchase tourism products and services on their own. To ensure that participants were familiar with the nature of Instagram, individuals without an active Instagram account were not considered In this way, lurkers were excluded. Altogether, the sample ultimately consisted of 182 respondents.

Fig. 1. Example of Instagram posts used in the survey.

### 4 Results

### 4.1 Demographic Profile of Respondents

Overall, as presented in Table 1, the majority of participants were female (75.3%) and skewed toward the younger population, aged between 18 and 29 (89%). In addition, 52.2% of participants agreed that they use Instagram to upload pictures and stories, while 47.8% only check the feed. The majority (61.5%) used Instagram several times a day.

Table 1. Demographic information (n = 182).

### 4.2 The Influence of Emoji on Emotion, Travel Intention, and Engagement

A non-parametric ANOVA Friedman test was chosen to determine the difference between participants' emotion when seeing Instagram posts with face emoji, non-face emoji, and no emoji at all. The result suggested a statistically significant difference between the levels of emotion. The Dunn-Bonferroni post hoc test (Table 2) showed that, even when comparing in pairs, the levels of emotion towards the three types of posts were all significantly different. To be more specific, the post with positive face emoji created the most positive emotions, while the post with non-face emoji provoked the least positive emotion. Hence, H1 can be accepted.

Likewise, the results indicated a statistically significant difference between the travel intention generated from the three types of posts (Table 2). Looking at the Dunn-Bonferroni test outcome, travel intention towards the posts with face emoji and no emoji was found to be statistically significant. Yet, there was no statistically significant difference when comparing travel intention towards the posts with non-face emoji and no emoji. Moreover, based on the mean ranks, the post with face emoji showed the highest mean value regarding travel intention. Thus, H2 is only partially supported since face emoji positively influenced travel intention while the appearance of non-face emoji showed no statistical difference regarding travel intention compared to Instagram posts with no emoji involved.

Concerning Instagram users' engagement, the Friedman test revealed a statistically significant difference for "like", "comment", and "share" when users saw posts with face emoji, non-face emoji, and no emoji at all. Turning to the results of the post-hoc test, a statistically significant difference could be found between the intention to "like" a post with face-emoji and with non-face emoji; people were most likely to like a post with face emoji, followed by no emoji and non-face emoji in the caption. However, the post hoc test presented no significant difference when comparing participants' commenting and sharing behaviour in pairs. The results indicated that the presence of face emoji and non-face emoji neither encouraged users to comment nor to share the post when compared to no emoji at all. Thus, hypothesis 3 must be rejected. Yet, based on the mean rank, it showed that people do indeed tend to comment and share posts with face-emoji the most, followed by posts with non-face emoji and no emoji at all.

Table 2. Result of Friedman and Dunn-Bonferroni tests.

### 4.3 Factors Influencing Travel Intention

Finally, in addition to the effect of emoji, Table 3 presents other external factors in a post that may influence one's travel intention. On the whole, the findings concluded that bright colours and saturated chroma ("the picture is beautiful") were the most compelling factor, followed by the caption of a post ("the caption of the post is interesting"). Similarly, when looking at posts with face emoji, brightness and chroma as well as the caption were also found to be influential factors.

Table 3. Instagram post factors influencing travel intention.

### 5 Discussion

Essentially, emoji add a positive meaning to the message (caption) [5], which, in turn, positively influences one's perception and instills a positive impression [30]. This study supports the appearance of (positive) face emoji in tourism-related Instagram posts promoting more positive emotions than non-face emoji. However, the result regarding the levels of emotions between posts with non-face emoji and without any emoji is not coherent with earlier literature claiming that non-face emoji also adds positive emotion to a post [5].

Meanwhile, face emoji in an Instagram post was also found to trigger one's intention to travel to a destination. This finding is supported by recent literature, suggesting that the presence of emoji on social media contributes to business development since it brings about a more favourable impression for the brand [11]. Subsequently, an increase in customers' willingness to purchase due to the inclusion of emoji echoes the context of buying hedonic products [8]. Thus, it can be concluded that the influence of positive face emoji on purchase intention also applies to destination marketing on Instagram. Face emoji create positive emotions for Instagram users, leading to an increase of their desire to travel; however, it is worth mentioning that not all emoji have the same effect. For instance, using non-face emoji in tourism-related Instagram posts generates a non-significant or less positive result than when no emoji is used. With regards to user engagement, users' liking, commenting, and sharing behaviours on Instagram have been recognised as crucial factors in promoting a destination [2]. In the same vein, this study's findings disclose that posts with positive face emoji attract the most "likes". This is consistent with the aforementioned results in which using positive emoji tends to generate the most positive emotion (e.g., joy, interest, and contentment), thereby obtaining the highest number of "likes".

On the other hand, since commenting requires more effort than clicking the "like" button, the more comments a post receives responds to a more profound level of user engagement [2]. When it comes to sharing, it also takes more time to consider sharing a post from DMOs since it reveals one's interest(s) to their friends, family, and other account followers [31]. This content can, in turn, influence other people's decision about a particular destination at a later stage [32]. Thus, the lack of significant difference regarding the likelihood that participants would comment or share the three types of posts shows that it is challenging to encourage Instagram users to comment or share Instagram posts posted by a DMO, despite the presence of emoji in the caption.

While the positive influence of positive face emoji on travel intention has already been examined, it is interesting to further investigate whether the desire to visit a destination is based solely on the effect of face emoji. First, in line with the literature, bright pictures with numerous colours activate a variety of senses, leading to higher popularity of a post [33]. That is, the findings suggest that the quality, colours, and brightness of pictures are also influential when it comes to travel intention. This result can also be explained through another recent study [34], stating that the brightness and colours of a picture affect viewers' emotion. Nevertheless, different from previous literature, where a post with a high number of likes and shares on Facebook can lead to higher purchase intention [35], our findings imply that the same notion may not be applicable to Instagram in the context of destination marketing. Regardless, all in all, the results support a compounding positive effect when face emoji is embedded in a caption along with "beautiful pictures" consisting of bright colours and chroma.

### 6 Conclusion

Until now, research on Instagram emoji has been limited and primarily focused on how emoji inserted in captions can influence post popularity [36]. Other researchers have mainly explored the impact of emoji in the context of promoting hedonic products [8, 9, 28]. This research thus adds knowledge to the existing body of literature related to semiotics and tourism. By connecting the effect of emoji to the domain of tourism marketing, this research reinforces that emoji may not only prompt a more positive impression towards a brand but also towards a place. Moreover, evidence revealed that, in the case of tourism marketing on Instagram, non-face emoji in captions may not bring about the same desired effect in generating positive marketing outcomes. This sheds light on a different perspective compared to existing research on how non-face emoji can also bring positive emotion when used in instant messaging [5]. In essence, this study contributes to the emerging field of marketing communications and semiotics in tourism research by identifying whether specific types of emoji (face emoji and nonface emoji) in tourism-related Instagram posts influence an individual's attention and behaviour.

The findings also offer numerous implications for destination marketers concerning emoji usage in digital marketing communications. Throughout this study, posts with positive emoji have performed better across all categories. Thus, marketers are advised to include emoji in their rhetoric strategy for social media marketing as it facilitates effective communication with potential travelers. Especially the usage of positive face emoji on Instagram can be beneficial for increasing positive impressions and feelings towards the promoted destination. However, destination marketers should also consider that different emoji serve different purposes. Since the findings show that the positive effect created by face emoji is associated with a higher intention of travel behaviour, it is advisable that for promotional posts aimed at increasing travel intention, preference should be given to positive face emoji over non-face emoji in order to guarantee more interactive and appealing content for DMOs and marketers. Furthermore, a picture's quality, including brightness and chroma, is another essential factor that marketers should pay attention to [2]. To conclude, in order to make the best out of an Instagram post, it is advised to include positive face emoji in the caption and high-quality pictures with bright colours.

Nonetheless, there are several limitations to this study that should be highlighted. First, the sample of the participants was unbalanced, with the majority being females. The sampling technique also resulted in an age skew. Even though a selection of images was utilized for this study, the biggest limitation lies in the fact that it is hard to control other stimuli and post elements. Moreover, considering the number of social media platforms available to date, the findings of this study are limited due to the focus placed on Instagram users.

For future research, scholars are advised to look at other types of hedonic advertising products. It is also recommended to investigate how different settings, advertisements, placements, and number of emoji influence tourists' online behaviour. Similarly, exploring which type of information source is more powerful in eliciting travel intention could be of interest for future research. Furthermore, scholars could study whether emojis can influence interest in information intensification. Future research could also zoom in on other social media platforms to investigate whether the same effects hold true regarding consumer behaviour and advertisement perceptions. Another interesting perspective for future research would be the comparison of emoji usage across different social media platforms. Finally, adding more face emoji and nonface emoji would additionally be a valuable direction to offer deeper insights.

### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Destinations

### **Navigation by Revealing Trade-offs for Content-Based Recommendations**

Linus W. Dietz(B), Sameera Thimbiri Palage, and Wolfgang W¨orndl

Department of Informatics, Technical University of Munich, Munich, Germany<EMAIL_ADDRESS>
**Abstract.** Conversational recommender systems have been introduced to provide users the opportunity to give feedback on items in a turn-based dialog until a final recommendation is accepted. Tourism is a complex domain for recommender systems because of high cost of recommending a wrong item and often relatively few ratings to learn user preferences. In a scenario such as recommending a city to visit, conversational contentbased recommendation may be advantageous, since users often struggle to specify their preferences without concrete examples. However, critiquing item features comes with challenges. Users might request item characteristics during recommendation that do not exist in reality, for example demanding very high item quality for a very low price. To tackle this problem, we present a novel conversational user interface which focuses on revealing the trade-offs of choosing one item over another. The recommendations are driven by a utility function that assesses the user's preference toward item features while learning the importance of the features to the user. This enables the system to guide the recommendation through the search space faster and accurately over prolonged interaction. We evaluated the system in an online study with 600 participants and find that our proposed paradigm leads to improved perceived accuracy and fewer conversational cycles compared to unit critiquing.

### **1 Introduction**

Nowadays, the algorithmic side of (RSs) research has reached an impressive maturity, such that it has become virtually impossible to tell which algorithms are objectively the best [1]. However, this improvement primarily applies to traditional RSs domains, such as e-commerce, movies, and to some extent music. For recommendations in complex domains, such as tourism, the algorithmic advances of the earlier decades are of lesser value. This is because there are insufficient ratings available, the items are not so well defined in terms of their scope, and it has also been shown that users demonstrate different decision making behavior compared to purchasing physical products [2]. These challenges necessitate employing sophisticated preference elicitation strategies, and instead of collaborative filtering algorithms, recommendations are often computed with a contentbased or knowledge-based paradigm. Given that traveling is a relatively rare, emotional, and high-stakes decision making scenario, RSs should provide users

c The Author(s) 2022

J. L. Stienmetz et al. (Eds.): ENTER 2022, *Information and Communication Technologies in Tourism 2022*, pp. 149–161, 2022. https://doi.org/10.1007/978-3-030-94751-4\_14

with the opportunity to familiarize themselves with the items in the domain and refine their initial preferences, since users often struggle to declare their true preferences [3]. For instance, recommending which city to travel, is a very good fit for the conversational, content-based recommendation paradigm, since there are no ratings available, despite the existence of several data sets [4, Table 2].

Conversational RSs allow a directed search through the item space using some kind of dialog between the system and user [5]. Early approaches, such as FindMe [6], allow users to critique certain aspects of suggested items, whereas more sophisticated approaches allow for compound critiques [7]. Based on the observation that critiques with concrete examples can be useful [8], we are astonished that not much attention has been paid to informing users about the tradeoffs involved in their critiquing choices. For example, many users would love to do a dream vacation to a buzzing city with outstanding cultural attractions, great food, a buzzy nightlife scene, favorable climate, at an affordable price tag. In reality, the combination of such features might be an empty set, thus, requiring compromising between conflicting preferences.

In this paper, we present a novel concept to navigate the item space that we call *"Navigation by Revealing Trade-offs."* The motivation for this combination of a novel user interface and a corresponding recommendation algorithm stems from the observation that conversational RSs tend to neglect informing their users about the trade-off involved in their critiquing choices.

After surveying the related work in Sect. 2, we present the user interface in Sect. 3, and describe the recommendation algorithms in Sect. 4. We choose the destination recommendation domain, as there are suitable data sets available and it inherently requires to make trade-offs between certain aspects of the trip. The experimental setup of a large-scale user study with 600 participants is described in Sect. 5 and we present the results in Sect. 6. Finally, we conclude our findings and point out future work in Sect. 7.

#### **2 Related Work**

In this work, our application domain is recommending cities for tourist destinations. As opposed to the recommendation of hotels or point of interests [9], cities as items have no meaningful ratings, thus, the user profile and items need to be matched based on elicited preferences and features of the items. To improve the user modeling, Neidhardt et al. [10] proposed a factor analysis for tourist roles and personality traits to reveal seven tourist behavioral patterns. The authors used a set of travel-related pictures, which were assigned to each of the seven factors by experts. Since the destinations were also characterized in the feature space of the Seven Factor model [11], they could perform content-based filtering for destination recommendation. Herzog and W¨orndl [12] proposed another travel RS where travel plans of multiple destinations satisfy user constraints such as budget and duration. The user modeling was done by binary indications of interest, i.e., check boxes, and the items were characterized using expert opinions and literature. Such expert-driven models are quite costly, thus, automated approaches are preferable to scale the item characterization. Prior approaches using mainly location-based social network (LBSN) data have been successfully employed in point-of interest recommendation [13] or to characterize cities [14]. The previously proposed city characterization approach [14] is based on the distribution of its venues, where a higher amount of venue relative to the city size leads to a higher scoring. The corresponding user study also suggested that unit critiquing is a fruitful approach in the destination recommendation domain. In this work, we re-use the prototype<sup>1</sup> and domain model of CityRec [14] to build a conversational RS.

Critiquing is a popular approach of eliciting and refining user preferences in a conversational manner. It is usually associated with content-based filtering, although there are some research incorporating collaborative approaches [15] or even unstructured item descriptions [16]. One of the early systems, FindMe [6] introduced the concept of unit critiquing that can be seen as the start of conversational exploration of the search space in RSs research. The static unit critiquing was quite successful in several domains [6,17], but there is opportunity to perform a smarter exploration of the item space [18]. For example, McCarthy et al. [7] proposed dynamic critiquing, to show how compound critiques can be generated dynamically, cycle-by-cycle by mining the feature patterns of the remaining products.

The evolution of dynamic compound critiques is the multi-attribute utility theory (MAUT) [19], which introduced a utility function to rank a list of multiattribute products. Once the user selects a critique, the corresponding product is set as the current preference product in the user model and a new set of critiques is generated using a utility function. The MAUT was successfully evaluated against dynamic critiquing [7] thereby reducing the number of critiquing cycles. Chen et al. extended the MAUT-based approach and called it "preference-based organization interfaces" [20]. In their approach, the authors organized all potential critiques in a trade-off vector showing whether the features were compromised or improved in comparison to the current recommendation. That enabled them to determine useful compound critiques and successfully evaluate it using a computer configuration data set. However, we feel that such an approach is more suited for products with clear specifications, since in tourism, relative differences between the features values of items are of higher importance.

One major issue with critiquing is the divergence of the intended direction of exploration. McGinty et al. [21] studied selection strategies for recommending items in critiquing. Their Adaptive Selection approach resulted in a reduction in critiquing cycles and they could prove that their critiquing-based approaches would converge faster than preference-based approaches. Another important insight of their work was that the user should not lose the progress, i.e., the previous recommendation should be included in the upcoming cycle.

Based on these observations, we introduce a paradigm to navigate the search space that we call *"Navigation by Revealing Trade-offs."* We propose a user interface element that visualizes the trade-offs involved in choosing one item instead of

<sup>1</sup> https://github.com/myftija/cityrec-prototype.

another in a less technical way than the preference-based organization interfaces by Chen and Pu [20]. Distinctively, our proposed interface gives the user an indication of the search space, i.e., where the current item's feature are located within the whole feature space, which was not given in the dynamic and compound critiquing approaches [7,22]. Furthermore, we used a utility function that determines the proposed items, aimed to resolve the *"wishful-thinking problem"* of users requesting item characteristics from the RS that do not exist in reality.

### **3 A User Interface Concept for Revealing Trade-Offs**

#### **3.1 Domain Model**

The pure content-based paradigm requires each item to be characterized along the same features to compute recommendations. In our case, we used an available data set of already characterized 180 cities all over the world [14]. This dataset comes with a score for each city in the categories of "Food", "Nightlife", "Arts & entertainment", "Outdoor and recreation", "Cost of living", "Shops and services", "Average temperature", "Average precipitation", and "Venue count". The domain of traveling successfully motivates our approach, since these features are natural in competition, i.e., a larger city with abundant cultural scene usually has higher cost of living, or, conversely, the nightlife options might be limited in small cities.

#### **3.2 User Interaction**

The user interaction through a web browser<sup>2</sup> goes through three major steps:

The contribution of this paper focuses on Step (2), the *Conversational Refining*. However, this key step must be seen in the context of the whole interaction design, which we now present step-by-step.

**Initial Preference Elicitation.** Before the user can start refining, an initial item needs to be determined. Ideally, the system would already have an established user profile based, e.g., through previous interactions. As we have no prior information about the user, we used a previously proposed approach to present the user with an initial seed of destinations where the user can select 3–5 [14]. This seed comprises of randomly selected candidates of various clusters. By this,

<sup>2</sup> The system is available under http://conversational-cityrec.cm.in.tum.de.

the diversity of the sample is warranted, as the user is presented with a representative set of items to choose from. Also, this method is quite fitting for the domain and the initial set of selected cities can directly serve as input for the utility functions of Step (2). We do not aim to evaluate this method from literature [14], as we used it in the same way in all experimental conditions.

**(a)** Preference Refining Page. This is shown to the user at the beginning of each conversational cycle. The current city is marked bold, and five alternatives are displayed on the spectrum of each feature.

**(b)** Trade-off Visualization. The green and red shades indicate the trade-off involved should the user choose Jakarta instead of Manila. The user explores various alternatives before continuing.

**Fig. 1.** User interface of navigation by revealing trade-offs. (Color figure online)

**Navigation by Revealing Trade-Offs.** Figure 1 shows the interface element for our conversational *"Navigation by Revealing Trade-offs"* approach. At the top of the page, the currently recommended city is shown; below is the novel user interface. This component shows the current city along with five other cities recommended based on the utility function. For each feature the five candidate items are shown in an ordered list from low to high depending on the score. Users can select an item to see the feature value differences in all feature spaces compared to the currently recommended city. An increase in feature value is indicated using a green shade, a decrease is shown in red.

If the user is satisfied with the current recommendation, the user can choose not to continue with refining, but to confirm the current recommendation. In this case, the user is forwarded to the final recommendation page.

**Final Recommendation Page.** This page shows the final recommendation to the user along with a survey to measure the performance of the recommendation approaches. The final recommended city is shown with details such as the city name, country and feature values.

**Baseline System.** To evaluate our proposed approach, we used a modified version of the "CityRec" destination RS [14], where one could critique features of several destinations to refine them by buttons indicating "much lower", "lower", "just right", "higher", "much higher". As the source code of this system was readily available, we used it as foundation for our experiments. We re-used the system architecture and the front-end for the initial Preference Elicitation page Step (1), and the Final Recommendation page in Step (3). However, notable differences in the user interface are that we did not use photos of cities to avoid bias due to the selection of images. Furthermore, we re-worked the unit critiquing algorithm to make it more comparable with our system. The critiques can be selected below using the same labels and logic for the adjustment as in the original approach [14], although, it is possible to adjust all features at once and the user is not limited in the number of critiquing cycles, thus, can refine the items until she is satisfied with the recommendation.

### **4 Algorithms**

Having described the user interface elements, this section presents the machinery that computes the recommendation and, therefore, directs the path the user takes through the search space. To enable reproducibility, the system and the study data set are available under an open-source license as a Dockerized software project on Github.<sup>3</sup>

#### **4.1 Cold-Start User Modeling**

Recall that in Step (1) of the system the initial input comprises a set of 3–5 items that are characterized along the aforementioned eight features. This already allows us to compute an initial user model by simply representing the user model as an eight dimensional vector with the mean feature values of the initial cities. Nevertheless, this method is quite simple and could be interchanged with any other strategy if more information about the user's preferences is available. Since this is not the case in our evaluation prototype, we used this simple method from literature.

#### **4.2 Candidate Selection Strategy**

The next step in the user interface requires finding candidates of which the user can choose one to progress the search for suitable recommendations. In typical content-based recommendation style, one could naively use any similarity metric, such as the Euclidean Distance on normalized feature values to compute some cities similar to the current user model. The top items can then be shown as alternatives to the user. One issue with this strategy is, that it does not consider the user preference variations during the refinement. Furthermore,

<sup>3</sup> https://github.com/LinusDietz/conversational-cityrec.

the convergence of the algorithm will be poor, since it presents the user with similar items to the current recommendation, thus, the user will not have the option to select a city with a significantly different feature value. Instead, we propose the "Variance Bi-distribution" utility function (Eq. 3) whose value is defined by two normal distributions per feature, each representing an increase of decrease of feature value. The two normal distributions are given as -N(μ1, σ<sup>2</sup>) and -N(μ2, σ<sup>2</sup>), where μ<sup>1</sup> and μ<sup>2</sup> define the position of the bell curves on the normalized value range of the feature, and σ defines the shape of the curve.

The distance between the currently selected reference item, ref*<sup>k</sup>* and the respective bell curves are computed by adding or subtracting an offset computed in Eq. 1. This offset is the standard deviation of each feature value f of all previous items in the conversational history H by the number of previous conversational iterations n. The numerator of the offset needs to be moderated by a constant C*m*, which we empirically determine for the dataset in Sect. 5.1. To summarize, the mean of the normal distribution is farther from the current user model if the variance of a feature is higher.

$$\mu\_1 = ref\_k - \frac{\sqrt{Var(f \in H)} \cdot C\_m}{n} \quad \mu\_2 = ref\_k + \frac{\sqrt{Var(f \in H)} \cdot C\_m}{n} \tag{1}$$

The second parameter of the normal distributions, σ, is computed in a similar way (cf. Eq. 2). This has the effect that with a higher variance, we obtain a flatter distribution and, thus, a lower impact of this feature on the utility score.

$$
\sigma = \frac{\sqrt{Var(f \in H)} \cdot C\_s}{n} \tag{2}
$$

The intuition behind this is that if the user has a strong preference regarding a feature having a certain value and consistently picks cities with a high temperature, the system is quite certain of this user's preference toward temperature and, thus, should put high weight to this feature. Conversely, if a user has selected cities with another feature having both low and high values resulting in a high variance, it can be seen as a signal that the user has no specific preferences toward the feature as it is not of importance to the user. Thus, the impact of such a high-variance feature should be smaller than a low-variance feature. Over time, we increase this effect by dividing through the number of previous iterations n. This further helps the algorithm converge.

The maximum score of the two distribution functions for a given item feature is taken as the utility score of the respective feature. We then compute the overall utility of each item as the sum of all feature scores of the utility function.

$$\text{utility} = \sum\_{f \in F} s(f) \tag{3}$$

**Convergence Behavior.** The effect of this utility function is that it balances exploration in the beginning and fine-tuning in later stages of the search. If a feature variance is high and the number of iterations small model adjusts μ<sup>1</sup> and μ<sup>2</sup> further away from the reference point, with a higher σ resulting in a flatter distribution of the feature's utility function. In this case, items far away in the feature space also would get higher utility scores, ensuring users are presented with cities more spread across the feature space. With a larger number of iterations, the user preferences for particular features are converging, i.e., the user will be presented with an increasingly narrower band of feature values to refine the preferences. As a result, μ<sup>1</sup> and μ<sup>2</sup> are closer to the feature value of the current recommended item, with a smaller σ, such that items with similar feature values have a substantially higher utility score than the cities with dissimilar feature values. However, if the variance of a feature is still high, the curve will stay quite flat giving this feature less weight, thus, recognizing that the user is rather indifferent toward this feature. This convergence behavior can be observed in Fig. 1. After some iterations, the algorithm determined that the user has a clear preference for high scores in the food and temperature aspects, and low scores in nightlife, outdoor & recreation, and cost. Thus, the refining candidates are quite close by each other, whereas they are spread along the spectrum in the arts & entertainment spectrum.

**Elimination of Candidates.** To further improve the convergence, we propose a variant that eliminates items whose feature values have been refined in a contrary way. The reasoning behind this elimination of candidates is that if a user refines a feature of an item, it becomes an explicit information that the value of the feature is unsatisfactory and should take only values toward the direction of the refinement. Thus, we can compute candidates just as before, however, items that have a lower (or higher) value than the original item ref*<sup>k</sup>* are removed from the search space. For example, if the user refines the value of *Arts & Entertainment* of Manila in Fig. 1b in favor of Jakarta, the system will assume that all cities that have a lower value in *Arts & Entertainment* than Manila should be excluded from future suggestions.

### **5 User-Centric Evaluation**

For the evaluation of the system, we chose a between-subject design to perform a large-scale online user study. First, we need to determine the constants of the Variance Bi-distribution Model for the current data set.

#### **5.1 Instantiation for the Domain**

During the development of the system, we noticed that using only the standard deviation divided by the number of iterations in Eq. 1 and 2, μ<sup>1</sup> and μ<sup>2</sup> would be too extreme, which will result in items recommended that are too far away from the current city. To moderate this effect, the constants C*<sup>m</sup>* and C*<sup>s</sup>* of Eqs. 1 and 2 were introduced for the Variance Bi-distribution Model. This step ensures an efficient navigation should be seen as an adjustment of the algorithmic properties to the data set at hand, as different domains can have different characteristics, i.e., a different number of items.

**Determining Constants.** The values of C*<sup>m</sup>* and C*<sup>s</sup>* can be determined in an offline setting using a simulation. This is because by systematically altering the values of C*<sup>m</sup>* and C*s*, we can see how quickly the algorithms converge from an initial setting after Step (1) to a desired item while making *consistent decisions*. In the context of the simulation, we define consistent decisions by choosing the item that is nearest to the target item using the distance metric of the RS. Thus, the simulator chooses candidates toward the target recommendation, just as a real user would, until that recommendation is part of the set of candidate items. For the cities, we used user interaction data to perform a realistic simulation [14]. The data set of 63 user sessions contained the initial city selections by the user and the final recommendation the user had selected. Having historic data for the simulation, we can now train the parameters using relevant scenarios, as opposed the randomized or exhaustive simulation strategies.

**Result.** Regarding parameters of the simulation, we varied C*<sup>m</sup>* from 2 to 6, and C*<sup>s</sup>* from 4 to 20, both in 0.5 intervals. For each these parameters' configuration, we recorded the session length of the 63 user sessions of the data set. The result of the simulation reveals a global optimum at C*<sup>m</sup>* = 3 and C*<sup>s</sup>* = 8.

#### **5.2 Online User Study**

We conducted the user study using the online experimentation platform Prolific.<sup>4</sup> We used a between-subject design and invited participants of the platform who had indicated "Traveling" as one of their hobbies. Only one independent variable was randomly assigned to the users, i.e., the critiquing system in Step (2). The three options<sup>5</sup> were the baseline unit critiquing system and the trade-offs UI using the Variance Bi-distribution Model without and with the elimination variant. As dependent variables, we used metrics about the user interaction and a subset of the ResQue Questionnaire (cf. Table 1), which is a validated, usercentric evaluation framework for RSs [23], where users indicate their agreement with each statement on a Five-point Likert Scale.

#### **6 Results**

The user study was conducted in December 2020 with 600 participants. Out of the 600 participants, we excluded 181 responses, which failed an attention check, showed very low interaction with the system, i.e., an interaction of less than 35 s, and did not use a desktop browser as instructed. This left us with 419 valid submissions (59.9% female, 39.1% male, 1% other) from 42 different countries. The users predominantly came from Europe, due to the time zone when the survey was initiated. The age distribution was 20.8% of below 21 year olds, 55.6% were 21–30, 13.1% were 31–40, 6.2% were 41–50, 3.1% were 51–60,

<sup>4</sup> https://prolific.co/.

<sup>5</sup> The variants can be tested under http://conversational-cityrec.cm.in.tum.de.

**Table 1.** Hypothesis testing of the dependent variables between the baseline unit critiquing and the two variants of the Trade-off refinement. The mean values of the survey items coded as integers from 1 to 5 are for informative purposes only.

and 1.2% were 61 years or older. With respect to the independent variables, 140 were assigned to the baseline unit critiquing, 130 to the Trade-off Refinement, and 149 to the Elimination Variant.

**Quantitative Analysis.** Regarding the number of conversational cycles, we observed that all sessions using the Trade-off interface were finished by the users within 6 cycles, with a mean value of 2.38/2.46, whereas the baseline unit critiquing interface needed more cycles with a mean value of 4.44 cycles. Thus, the Trade-off UI reduced the iterations by of 46.4% (44.6% in the elimination variant), which is a significant reduction when testing the hypothesis using a t-test (cf. last row of Table 1). Note that the user interface was set up in a way, so that at least one interaction cycle had to be performed, before the users could accept the current recommendation as final result.

For the survey items, we computed cross-wise Wilcoxon rank sum tests for independent populations using the three independent variables. The null hypotheses were that there is no difference in the median of the responses. Since we could not find significant differences between the Trade-off refining and Trade-off refining with the Elimination variant, we only tabulated the outcomes in Table 1 with respect to the baseline unit critiquing. Besides the analysis of the number of conversational cycles, we could refute the null hypothesis in favor of the Trade-off Variants in (Q1) and (Q9), while the baseline received better responses in (Q6), (Q7), and (Q8). This mixed result can be summarized in a way, that the Trade-off interface had superior perceived recommendation accuracy at the expense of the users' perceived ease of use.

**Discussion.** The superior perceived accuracy measured by (Q1) at about 45% fewer conversational cycles, underlines the merit of our proposed user interface. However, the subjects rated the usability-related metrics of the unit critiquing system higher (Q6–Q8). We suspect that this due to that unit critiquing has already been employed in various RSs, so it is quite possible that many users were already familiar with this concept. Dealing with a new refinement interface involving reasoning about trade-offs certainly involves more cognitive effort and, thus, might need more familiarization (Q8) than only one session. The study was designed in a way that users could only submit the survey once and we did not familiarize the users with the system before their session to avoid learning effects. The significant difference in (Q9) "This recommender system influenced my selection of cities." in favor of the Trade-off interface is likely an artifact of the comparative lengthy search in the unit critiquing, since both values are in the center of the Likert Scale. Interestingly, there were no significant differences in any dependent variables between the Trade-off refinement and its Elimination variant. We attribute this to the low number of conversational cycles that were needed to come up with a satisfactory result. In the given data set of 180 cities, the elimination of candidates was probably not necessary, as the utility function was able to recommend attractive items after two or three cycles. Nevertheless, we are confident that the concept of elimination of parts of the search space based on the users' choices could be useful and we plan to analyze the merit of the Elimination variant with larger item sets of over 1000 items.

#### **7 Conclusions**

The success of modern recommender systems depends on the seamless integration of algorithms and user interface elements. Given that existing critiquing systems have often neglected to explicitly inform users about the trade-offs of the critiquing actions, we developed the Navigation by Revealing Trade-offs system, which integrates a user interface concept with a utility function to compute refinement candidates. The evaluation shows that perceived accuracy is better than the unit critiquing baseline at similar reductions in the number of conversational cycles as other advanced critiquing approaches have demonstrated [19,21].

Based on this promising result, further analyses of this refinement paradigm should follow with larger item sets to analyze the merits of the Elimination variant. Since our study followed a between-subject design, we also can not answer whether the higher ratings for the interface adequacy are due to that unit critiquing being conceptually easier to understand or users are more familiar with such a long-established paradigm. Therefore, the usability and learnability should be investigated in a usability analysis in a controlled laboratory setting.

### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Multisensory VR Experiences in Destination Management

Barbara Prodinger(&) and Barbara Neuhofer

University of Applied Sciences Salzburg, Puch/Salzburg, Austria {bprodinger.imte-m2019<EMAIL_ADDRESS>
Abstract. The rapid development of information and communication technologies (ICTs) and the high level of consumer acceptance have made it increasingly complex to retain loyal customers. Virtual Reality (VR) has become a solution that allows tourism providers to design technology-enhanced experiences along the entire customer journey. While most VR offers focus on pre-travel experiences, the potential of VR in the post-travel phase is still little explored. Considering that multisensory tourism experiences contribute to memory formation, the multisensory extension of VR (4D VR) in post-travel experiences is of interest. Thus, through a quantitative field experiment, this study aims to detect what effect the stimulation of different senses during the use of VR has on the overall experience and how this influences the brand relationship quality. The results revealed elevated levels of technology acceptance, which consequently enhances the traveler's overall VR experience. The multisensory component positively affects one realm of an experience in the area of escapism and thus correlates with the overall experience. However, there is no significant difference between 3D and 4D regarding the level of brand relationship quality. The study expands the literature on 4D VR experiences and supports tourism practitioners in the implementation to strengthen the relationship between a destination and its guests.

Keywords: Multisensory virtual reality Brand relationship quality Experience economy

### 1 Introduction

Attaining and maintaining loyal customers has become a progressively complex challenge in today's market. This can be attributed to the widespread adaptation and acceptance of various ICTs, which customers engage with the brand [1]. One developing sector of technology progression is VR [2]. VR is expected to revolutionize tourism experiences by providing extensive sensory information [3] representing a source of education, entertainment, information, planning, heritage preservation and accessibility [4] and thus, improving customer experiences along all phases of the customer journey [5, 6].

The evolution of VR is based on the desire of people to escape their physical space. This aspiration is to some extent related to travel motives, such as the urge to be temporarily able to leave one's familiar surroundings [7]. In this regard, virtual tours of destinations are becoming an increasingly popular approach for tourism marketing [3] and alter how tourism services and products are promoted and consumed [8]. The number of organizations that incorporate VR activities in their marketing is growing, although the implementation of VR is still nascent [6, 9].

Fully immersive virtual touristic experiences that trigger the user's visual and potentially other senses are enabled by isolating the user from the real world into a virtual environment through 360-degree real-life or synthetic content within a non-, semi-, or fully immersive VR system [4]. For tourism businesses, it is suggested to integrate at least two more senses in the use of VR [3, 10]. Instead of 3D VR, which is reduced to auditory and visual sensory stimuli, multisensory VR (4D VR) provides DMOs with an opportunity to present their services in a more engaging way [3, 4]. The multisensory approach in the context of sight, hearing, touch, smell and taste intensifies emotions and influences the overall VR experience. It does not only better appeal to the customer, but the combination also enriches the perceived value, encourages the actual consumer behavior and decision-making process [10].

The growing field of VR demonstrates the need for further research in the context of tourism [4]. Towards this aim, previous studies focused on the effect of VR technologies on the pre-travel and on-site phases [4, 5]. Nevertheless, the investigation of the post-travel phase is lacking [4]. The post-travel phase is however of major interest, as tourism organizations aim for elevated Brand Relation Quality (BRQ) by triggering post-experience related memory and relationship formation. To achieve a higher level of BRQ, VR experiences need to be authentic, personalized and relevant.

This study thus takes an integrated approach, to examine not only VR in the posttravel phase, but also to assess the effect of technology acceptance and a multisensory VR experience on the levels of BRQ. The following research questions have been identified:

### 2 Literature Review

### 2.1 Experiences as Value Propositions

Over the last years, less attention has been paid to products and services, due to the significant shift from service-based to experience- based value propositions in tourism and hospitality [11]. Creating positive emotions and long-term memories has become the main focus [12]. Consumers want to be engaged in an inherently personal way and want to have a good time while enjoying memorable experiences [13]. Additionally, ICTs are nowadays an integral part in the way tourism experiences are designed and consumed [14]. While experiences are multidimensional constructs, the factors of entertainment, education, escapist and esthetics need to be considered when designing experiences. All four realms form the "sweet spot" [15].

### 2.2 Technology Acceptance

The consumer's willingness to adopt new technologies and to deal with different systems and devices that have not been used before, is crucial for the success of new applications [16]. Factors that strongly influence the acceptance of technologies have been researched across multiple ICTs and user populations [17], whereas Davis' [18] Technology Acceptance Model (TAM) has been widely validated because of its robustness, generalizability and explanatory power by various scholars in the past [19]. The variables perceived usefulness and the perceived ease of use of a certain application comprise the user's technology acceptance [18]. Customers with a high degree of technology acceptance immediately adapt to new products and are to know their features [20]. This leads to the following hypothesis:

H1: Participants with a higher technology acceptance have a better overall VR experience.

### 2.3 Multisensory VR in Tourism

Latest research in the field of VR is grounded in the human desire to escape the boundaries of the real world and experience new places by embracing the cyber world [20]. Virtual tours of destinations are becoming an increasingly popular approach in tourism marketing [3] and have altered the ways in which tourism services and products are promoted and consumed [7]. VR is often labeled as the future of tourism marketing [21]. Tourism can benefit from VR in the areas of entertainment, education, management, planning, heritage preservation [2, 22] or accessibility before, during and after travel [4], enriching the customer experience along the customer journey [5, 6]. With the possibility to rotate 360-degree around a certain angle, VR enables tourists to experience a destination before the actual consumption [3]. To echo the travel experience and review the journey after the departure, travelers can share 360-degree-videos through VR with family and friends at home. This inspires other people during their pre-trip inspiration phase [4]. It is said that VR has a stronger influence on the desire to visit a destination as other promotional materials [23], which contradicts Tussyadiah et al. [8] who emphasize that fully immersive VR experiences are less powerful than traditional travel guides.

One of the main recent developments has been the shift from 3D to 4D multisensory VR [3]. The nature of a tourism experience is intangible, yet physical and multisensory [24]. To allow an experience that is close to the real one, it is suggested to engage all five human senses [25]. It does not only better appeal to the customer, but the combination also enriches the level of immersion, perceived value, and encourages consumer behavior and decision making [10]. Multisensory VR provides DMOs with the opportunity to present their services and products in a more engaging way [3], as the multisensory approach in the context of sight, hearing, touch, smell and taste intensifies emotions linked to the overall experience [10]. Offering the user unexpected sensory input, such as audio or a new scenery, takes the person off the cognitive track [26]. This results in the following hypothesis and sub-hypotheses:

H2: Multisensory stimuli enhance the user's overall VR experience.

H2a: Multisensory stimuli enhance the educational realm of the user's VR experience.

H2b: Multisensory stimuli enhance the entertainment realm of the user's VR experience.

H2c: Multisensory stimuli enhance the esthetic realm of the user's VR experience.

H2d: Multisensory stimuli enhance the escapist realm of the user's VR experience.

### 2.4 Brand Relationship Quality

An essential strategic factor in the long-term relationship between the brand and its customers, is customer retention in brand management [27]. Relationship quality itself inherits the perception of another party regarding social, cultural, economic, or political aspects [28]. To debate the importance of long-term relationships, the concept has been applied in various fields. In the tourism industry, entities are realizing the need for an in-depth understanding of the established bonds between travelers and the destination [29], because a tourist's commitment to a destination defines the competitiveness of a region, which is considered as a pillar of brand quality [30]. Marketing-based VR could play a critical role in facilitating a brand's performance, whereas negatively perceived VR could diminish the brand relationship [30].

There are multiple constructs and factors measuring BRQ [32]. As BRQ evolves through a meaningful interaction between a brand and its consumers, actions shape the quality of the relationships by fostering, diluting or even dissolving them [33]. The strength of the brand relationship is discussed by six indicators, including: "love and passion, self-connection, commitment, interdependence, intimacy and brand partner quality" [33, p. 366].

Brand love is part of the multidimensional attachment construct, including the items passion, affection and connection [34]. Not consuming the brand for a while is experienced as "something is missing" [33]. It is not only based on passion, but also on frequency and an ongoing long-term relationship [34]. Within the facet of selfconnection, the brand sends out important identity values, themes and tasks, communicating and stating an aspect of self. Commitment is the psychological or emotional attachment to a brand and its product class [33]. This develops more from the person's attitudinal rather than behavioral trait [35], preferring one product or service over another. Intimacy arises through the customer's elaborate knowledge about the company and its personal brand stories [36]. By advertising cues, such as slogans or a specific character, the personalized brand information enters the long-term memory [37]. Brand partner quality is the positive alignment of the brand towards its customers. People should feel respected, listened to, wanted and cared for, building the base for the customer's judgement about the brands performance [33]. Therefore, the following hypothesis and sub-hypothesis have been formed:

H3: A 4D VR experience has a stronger influence on the levels of BRQ than a 3D experience.

H3a: A multisensory VR experience in the post-travel phase will increase the love and passion of travelers towards a destination.

H3b: A multisensory VR experience in the post-travel phase will increase the selfconnection of travelers towards a destination.

H3c: A multisensory VR experience in the post-travel phase will increase the commitment of travelers towards a destination.

H3d: A multisensory VR experience in the post-travel phase will increase the interdependence of travelers towards a destination.

H3e: A multisensory VR experience in the post-travel phase will increase the intimacy of travelers towards a destination.

H3f: A multisensory VR experience in the post-travel phase will increase brand partner quality towards a destination.

### 2.5 Conceptual Framework

The following hypothesis model designed for this research indicates, that the overall VR experience is dependent on Pine & Gilmore's [15] four realms of an experience formed by education, entertainment, esthetic and escapist factors. The VR experience depends on the user's technology acceptance, resulting from the user's perceived usefulness and perceived ease of use [18]. The influence of the overall VR experience on the six levels of brand relationship quality concludes the construct (Fig. 1).

Fig. 1. Conceptual framework based on Pine & Gilmore [15], Davis [18] and Fournier [33].

### 3 Methodology

The study is based on an experimental research design. Laboratory experiments are commonly chosen as a methodology to explore causal knowledge in life and physical sciences but have been criticized for their lack of generalizability and realism. In contrast, field experiments are logistically more difficult, time-consuming and costly [38]. To enable the quantitative field experiment by examining a DMO's real use of VR, the tourism destination "Wagrain-Kleinarl" has been chosen as a case study region. To expand research on the use of VR in the post-travel phase, participants of the study have been guests who spent their vacations in the destination in the past. In the research process, study participants were recruited via the destination's Facebook page which has more than 15,000 followers and Instagram channel with over 9,300 subscribers. Within one hour, the limited number of participants, willing to be part of the project, was achieved. In total, 100 guests of the destination agreed to participate in the field experiment.

All participants were randomly assigned to Group A, the experimental group, and Group B, the control group. Within the framework of a post-test-only control group design, both groups did not conduct a pre-test and underwent the same visual and auditory VR experience. The goal was to add a different intervention to only the experimental group, in this case an olfactory and haptic component to the VR experience. An online questionnaire was utilized as a measurement tool. Levitt and List [39] pointed out that the same degree of control as in laboratory experiments cannot be achieved in field-experiments. For this reason, all other factors that could influence the outcome must be controlled [40] and possible confounding variables detected [41].

For this study's research design, a prototype of a multisensory VR experience, tailored to the destination's corporate identity was sent with a package to the guests' homes. Both groups received a box, including a pair of VR cardboard glasses, detailed instructions on how to conduct the VR experience, a QR code leading to a tailored 360-degree-video hosted on YouTube and a second QR code guiding to the online survey. The VR video was shot with an Insta360 in April 2021. Within 3:18 min, the user experienced a sunrise above the destination of Wagrain-Kleinarl, a farm, a stable with two children playing in the hay, a mountain and lake scenery. In addition, the experimental group received the olfactory and haptic component of hay and herbs from the region as intervention. In the first step, the participants were instructed to sit down in a quiet place. The second step was to take the VR cardboard goggles out of the box and open the flap for the smartphone. After that, the participant scanned the QR code on the instructions, whereupon a video opened. After selecting the VR mode and turning the device to landscape, the user placed the smartphone in the goggles. Additionally, the experimental group was instructed to pick up the hay and take a deep breath. After both groups watched the same 360-degreevideo in the goggles, the QR code for the online survey was scanned.

Both groups conducted the same questionnaire, measuring the user's technology acceptance in case of perceived usefulness, perceived ease of use and attitude towards using [18, 20]. This allows a better understanding of the tourist's use of the virtual environment, which in return influences travel intentions [19]. To measure the experience dimensions within the virtual environment, Pine and Gilmore's [15] four realms of an experience, i.e. entertainment, educational, escapist and esthetic, have been adopted as items. Fournier's [33] construct of BRQ explored the six factors of brand partner quality regarding the traveler's intimacy, commitment, self-connection, interdependence as well as love and passion towards the destination. The online survey was a fully standardized questionnaire, based on predominantly closed questions with suggested answers and statements to assess on a Likert scale ranking 1 "totally agree" to 5 "don't agree at all". Subsequently, the data analysis was executed by using SPSS20.

## 4 Results

### 4.1 Sample Description and Reliability Analysis

The tested sample consisted of 100 participants, whereas a response rate of 85% (n = 85) has been achieved. 41 of those participants belong to the experimental group and 44 to the control group. The majority of the participants were female with 65.9%. The age groups were well distributed: 34% were under 31 years old, 31.8% between 31 and 40 years and 34.1% over 40 years old. 50% of the questioned travelers were Austrians, closely followed with 40% German guests. In total, six European nations were represented. The highest education of 30% of the participants is high school, followed by Master's and Bachelor's degree. With 60% of the participants of the study are currently employees. By investigating the reliability, the stability and consistency of each constructed was evaluated. No statistical differences between the control and experimental group were identified. The dimensions of technology acceptance reached a Cronbach's Alpha from .609 to .756, the four dimensions of the overall VR experience achieved values from .627 in education and the highest value of .902 in the realm of escapist. All six dimensions of BRQ have a Cronbach's Alpha over .709.

### 4.2 Test of Normality and Hypotheses

With the Shapiro-Wilk and Kolmogorov-Smirnov test, the questionnaire was tested for normality. A normal distribution was present in H2, H2a, H2c and H3d. A T-Test was used to compare the two samples. For the remaining hypotheses, the U-Test according to Mann and Whitney test was generated. Regarding the hypothesis testing in Table 1, the questionnaire applied a 5-point Likert scale with 1 representing the best rating and 5 the worst, showcasing with values 2.5 the more positive answers. In the following table, the hypothesis testing is presented. With significant values from .29 to .91, H3a to H3f had to be rejected and are therefore not separately presented in in the table below.

Table 1. Hypotheses testing

\* Rejected by T-Test

### 4.3 Further Correlation Analysis

Examining the technological acceptance in correlation with the fact that the respondents have a VR experience, no significant correlations were found (p = 0.096). Thus, it can be said that technology acceptance does not depend on prior experience of using VR. A highly significant result was shown by the Kruskal Wallis test (p = 0.002). Accordingly, it can be said that technology acceptance is related to age. The youngest age group (>31 years) had a significantly higher technology acceptance compared to the eldest respondents (>40 years). Differences were not only apparent in age. There were also significant differences (p = 0.036) in technology acceptance regarding the highest level of education completed. In the middle ranks, it could be observed that those with a bachelor's degree have a significantly higher technology acceptance than those with a completed primary school education. Furthermore, the T-test showed that participants with prior experience of VR did not enjoy a more intense VR experience than those who had never experienced VR before (p = 0.692). Thus, the overall VRexperience was not related to whether someone had experienced VR before. The T-test which was used to evaluate whether a difference occurs with respect to the overall VR experience, indicated that there was no such difference (p = 0.544) regarding distractions, such as noise from a radio or TV. Upon further review, it appeared that other people in the room in fact do not make a difference on the VR experience (p = 0.425).

### 5 Discussion

Previous research stated that the success of new applications depends on the willingness of customers to adopt them [16]. In this study, the openness towards new technologies was elevated among the participants with a mean value of 1.4. Both groups indicated a relatively high technology acceptance regarding the perceived ease of use and usefulness of the VR application. With a positive correlation according to Pearson (0.602), the assertion proves to be true. People with a high technology acceptance enjoyed a greater overall VR experience. The study's findings differ from the literature, as the participants' prior experience with VR did not influence the user's degree of technology acceptance [42].

In the design of VR experiences, Hopf et al. [3] suggested, that at least two more senses should be involved in the use of VR. In this study, no major deviation between the purely audio-visual experience and the advanced experience with the haptic and olfactory components was detected. Although the means demonstrated, that 4D improves the VR experience, according to the T-Test, statistically significant differences could not be found in this data. Therefore, engaging additional senses besides visual and auditory do not enable a better overall VR experience. Reflecting deeper on the overall VR experience, a rich experience consists of all four realms [15]. Here it can be stated that the additional multisensory components allowed a greater degree of escapism compared to a 3D experience. Contradictory results were similarly obtained for education and esthetics. Solely the area of entertainment, with equal mean values and a U-Test after normal distribution showed, that the hypothesis had to be rejected.

This study also explored the effect of a multisensory experience on the levels of BRQ. Findings revealed, that with a score of 0.916, there was no significant difference between a 4D and a 3D experience, even though each level represented elevated levels. Regarding the most important aspect of BRQ, love and passion towards a brand [43], all items presented very good to good mean values. The study supports Fournier [33] by declaring, that "something is missing" when one cannot consume a brand for a certain period of time. The multisensory component did not lead to any deviations.

### 6 Conclusions and Implications

This study aimed to explore the effect of multisensory VR on the levels of BRQ in destination management. By considering the user's degree of technology acceptance, this study supports prior research in agreeing to the positive contribution to the overall VR experience. The study contributes to the experience economy, stating that experiences are multidimensional. Specifically, the engagement of the senses contributes significantly to intensity and authenticity [25]. Applying the four realms of a memorable experience as a benchmark proved to be useful. It was confirmed that the haptic and olfactory components positively influence the audio-visual VR experience in the area of escapism. However, this research clearly revealed that the additional sensory stimuli do not enhance the overall VR experience in the post-travel phase, although guests did feel more immersed in the virtual world. Thus, there is no different impact on the relationship between the guest and the region when more than audio-visual senses are engaged in a virtual reality experience. One possible reason for this could be that, in general, a VR experience in the post-travel phase is very novel to customers and thus this single factor of the multisensory component does not entail a significant difference on the BRQ.

In terms of theoretical implications, this study opened a novel research area by bridging multisensory VR experiences, post-travel experiences and BRQ. The study partly contradicts previous research highlighting the importance of the implementation of additional senses in the design of VR experiences. Limitations have been discovered, since the experiment did not take place in a laboratory setting under observation, unmediated confounding variables, such as noise, other odor influences or social presence from other individuals or subjects could have influenced the results. Due to the choice of the post-test only control-group design, no pretest was conducted. For further research it is suggested to apply different objective and subjective measurement techniques, such as personal observations or post-experiment interviews to verify the results and to gain further insights into the traveler's perception of multisensory VR in the post-travel phase. The same prototype could be tested to explore emotional arousal or booking intentions. The study provides implications for tourism providers in the design of multisensory VR applications. Herewith, the implementation of the four realms education, entertainment, esthetics and escapism is recommended. Especially in the fourth variable, the additional multisensory components allow for a stronger immersion. Ultimately, the participant's high technology acceptance revealed that multisensory VR applications in the post-travel phase are highly appreciated by the destination's guests.

Acknowledgments. This work was supported by the Austrian Federal Ministry for Digital and Economic Affairs and the Destination Management Organization Wagrain-Kleinarl Tourism, which funded the field experiment.

### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Do DMOs Promote the Right Aspects of the Destination? A Study of Instagram Photography with a Visual Classifier

Lyndon Nixon(&)

School of Applied Data Science, MODUL University Vienna, Am Kahlenberg 1, 1190 Vienna, Austria<EMAIL_ADDRESS>
Abstract. As global travel emerges from the pandemic, pent up interest in travel will lead to consumers making their choice between global destinations. Instagram is a key source of destination inspiration. DMO marketing success on this channel relies on projecting a destination image that resonates with this target group. However, usual text-based marketing intelligence on this channel does not work as content is consumed first and foremost as a visual projection. The author has built a deep learning based visual classifier for destination image measurement from photos. In this paper, we compare projected and perceived destination images in Instagram photography for four of the most Instagrammed destinations worldwide. We find that whereas the projected destination image aligns well to the perceived image, there are specific aspects of the destinations that are of more interest to Instagrammers than reflected in the current destination marketing.

Keywords: Destination image Visual classification Instagram photography

### 1 Introduction

Instagram may be the most significant source of destination inspiration for a significant number of travel-interested Internet users, especially in the 18–34 year old age group<sup>1</sup> . After all, 48% of Instagram users rely on the images and videos that they see on the social media platform to inform their travel decision making and 35% of users use the platform to discover new places<sup>2</sup> . It is therefore not surprising that destination marketing organisations (DMOs) have expended significant effort in marketing on this channel. Instagram is a highly visual social network and therefore content marketing is focused on the use of photography and videos, more so than the text of postings. When an Instagram user consumes destination marketing content, the intention of the

<sup>1</sup> 62.8% of the users are aged 18–34 according to the Digital Internet Snapshot 2021 https:// datareportal.com/reports/digital-2021-global-overview-report (last accessed on 24 March 2021).

<sup>2</sup> Rezab, J. "Instagram: The Place To Be For Travel Brands". Available online. https://www.mediapost. com/publications/article/263167/instagram-the-place-to-be-for-travel-brands.html (last accessed on 28 July 2021. Requires free sign-in.).

<sup>©</sup> The Author(s) 2022

J. L. Stienmetz et al. (Eds.): ENTER 2022, Information and Communication Technologies in Tourism 2022, pp. 174–186, 2022. https://doi.org/10.1007/978-3-030-94751-4\_16

marketer is that the visually projected image of the destination resonates with them and increases their interest and positive attitude towards the destination – leading to an intent to visit the destination. Multiple case studies have backed up the assumption that an optimally marketed destination on Instagram can lead to an increase in visitor numbers<sup>3</sup> . Indeed, Instagram is now being blamed for causing overtourism in certain "Instagrammable" locations<sup>4</sup> . A challenge for DMOs or other stakeholders is to know if they are indeed marketing their destination optimally on Instagram. A rise in visitor numbers can only suggest marketing success a posteriori. Marketing intelligence – the analysis of marketing data to identify the factors of marketing success – has classically made use of analysis of text (of reviews, or of social media postings) and quantifiable metrics (e.g., in social networks number of likes, comments and shares) [15]. However, the primary component of Instagram marketing is visual: the photo or video used in the posting. Ignoring this component in marketing intelligence in Instagram would mean the primary factor for the marketing success is being excluded from the analysis.

In this paper, our assumption is that the target audience for the destination marketing – the Instagram users – reveal the aspects they are most interested in and find most appealing in a visit to the destination through their own choice of the photography they post to the social network as part of their touristic experience. Photography has been long understood as an expression of the photographer's mental image of the destination [19]. They take photos of the aspects they consider most indicative, and this is now even more the case in social networks where the user chooses which of their photos they want to post and share with others. Therefore, the content of Instagram photography from a destination (user generated content, or UGC) can be analysed for the "perceived destination image". Equally, the content posted by the DMO on its Instagram channel can be subject to the same analysis to measure the "projected destination image" [11]. Common marketing theory says that marketers want to influence consumer attitudes towards their product and can measure the success of this influence by surveying consumers after a marketing campaign and determining if attitudes have become closer to the marketing message. In e-tourism, this has translated as marketers defining their intended "projected destination image" and measuring how close the "perceived destination image" comes to it after a marketing campaign. However, such thinking persists the assumption that marketers are the primary conveyors of the destinations image when in fact today's traveller has a destination image which is strongly influenced by the UGC they consume about that destination [14], especially from social networks [25]. Therefore, we turn this theory on its head and argue that, in fact, destination marketers should inform themselves of the "perceived destination image" of their target audience and, given that this reflects the that resonate most with those consumers, aim to align their "projected destination image" to it.

Measuring destination image from Instagram photography is quite different from analysis of text or statistics. In the following chapter, we look at destination image in e-

<sup>3</sup> e.g. Iceland and New Zealand case studies from "Insta tourism explained: What, why and where". Available online. https://tourismteacher.com/insta-tourism/ (last accessed on 28 July 2021).

<sup>4</sup> Tenbarge, K. "15 destinations Instagram has helped ruin" Available online. https://www.insider.com/ travel-destinations-instagram-influencers-ruined-2019-7 (last accessed on 28 July 2021).

tourism research and how visual media classification has been researched in the field of computing. Section 3 introduces the author's implementation of a deep learning-based visual classifier for measuring destination image and its evaluation results for accurate classification of tourism photography. Section 4 explains our experiment: we take four top Instagram destinations and download sets of DMO and UGC photos. We describe the measured destination image as multidimensional vectors and compare projected and perceived images including for selected features. We identify differences between the destination marketing and UGC. Section 5 concludes this paper with a look at our findings, what DMOs can learn from this for their future destination marketing and which limitations and future work remain.

### 2 State of the Art

Destination image has been a subject of tourism research since several decades [5]. It has been defined as the mental construct a person has of a destination and is sourced initially from indirect sources, i.e., before they visit the destination and form an opinion [8]. From the perspective of a group, the destination image may be considered the ideas and feelings most expressed about the destination (e.g., Costa Rica may be most typically associated with jungle and adventure). Destination image has been important as a concept to tourism stakeholders as a correlation is assumed between a positive image and an increased intention to visit [16]. Originally, destination marketers considered their marketing as significant in determining the image of their destination among a target audience, as their marketing materials were a primary source of impressions about the destination for someone who was exploring the option to visit. Nowadays this no longer holds as the Web and digital photography has led to an explosion of destination imagery from tourists which is globally distributed and easily consumed on popular Websites. This imagery is now dominant in forming an initial destination image among connected consumers [13]. Destination marketers now need to accept that Internet users participate in the co-creation of the destination brand [1], referring to the influence on the destination image of the user content being shared online [2].

Initial destination image research used either surveys and other solicitation techniques from the public or expert knowledge to define the attributes people commonly associate with destinations [8]. Measurement of destination image consists of determining a value for each attribute. Given a classification of those attributes, surveys were also used to directly extract the destination image for a given destination from a group of people, e.g. [3]. As more destination related content became available online through Websites, researchers turned from survey-based approaches to data-based approaches where they collected the data from the Web and used software tools to analyse it. Text analysis could be used with collections of reviews of tourism POIs (points of interest), e.g. [15]. However, this proves less effective with multimedia content (images and videos) where the only text that might be available would be an associated description, tags or even the content of the comments on that multimedia item.

Tourism researchers have long been aware of the importance of photography in understanding how visitors perceive a destination [6]. The choice of elements in the photos are seen as reflecting the photographer's own idea of what is most important about the destination [19]. Initial research on destination image and photography solicited directly from participants their impressions from observing a certain photo, a technique known as PEI (Photo Elicitation Interview), which has also been applied with Instagram [9]. Later work manually classified the photos found on Websites according to their content or theme, e.g. [21, 22]. Such efforts were largely small scale and did not concur in how to classify the photos, nor aligned their classifications to other research on destination image attributes [20].

With the advent of social networks and photo sharing, the scale of available photography about destinations has expanded greatly. The Web also drove "big data" research where analysis could be performed on larger scale data collections that could be sourced online and hence provide more accurate results. However, in the research domain of image understanding, accessible and accurate software systems have only emerged with the advent of deep learning-based approaches [24]. These use neural networks (AI systems that seek to replicate the neural activity of the brain) with multiple processing layers (hence "deep") to 'learn' to identify the content of images based on training with large datasets of already classified images. State of the art deep learning classifiers have been trained with the ImageNet dataset, a set of 1.3 million photos annotated according to a list of 1 000 visual categories, reporting high levels of accuracy when evaluated against previously unseen image sets<sup>5</sup> . The classification abilities of deep neural networks are also available to users via Web based APIs hosted by companies such as Google (Cloud Vision), IBM (Watson Visual Recognition) or Microsoft (Azure Computer Vision). They tend to offer object recognition, i.e., classification of images based on the detection of visible objects, drawing from concept sets in the tens of thousands.

Tourism research using deep learning classifiers to analyse larger photo sets has only appeared in the past few years, cf. [7, 12, 26–28]. Off-the-shelf pre-trained classifiers have been used which return generic concepts for each image, largely based on the ImageNet classification. A few papers have acknowledged that this is not directly useable in the touristic context where destination image is defined as an aggregation of more general cognitive attributes (such as "entertainment") rather than the specific objects explicitly visible in the image (such as "guitar", "turntables" or "stage"). The classifiers are not trained specifically on tourist photography and the few experiments to measure the accuracy of a classifier when used in the tourism domain report lower figures than for the ImageNet classification for which evaluation results are reported. [18] evaluated two pre-trained classifiers with Instagram photography of Vienna and found f1 measures (the harmonic mean of precision and recall) of 0.54. [12] found an accuracy score of 28% for their classifier when tested specifically in the use case of touristic destination image. Past work has also not mapped the labels

<sup>5</sup> https://keras.io/api/applications/ lists pre-trained models available in the Python library Keras with top-5 accuracy scores of 89 to 96%.

returned by the off-the-shelf classifiers to destination image attributes, a task made more complex by the lack of any official listing of all labels.

There is the option to train a deep learning neural network specifically for visual classification in some new domain, given that the ImageNet pre-training is accepted to be too generic when classification is to be done for a specific use case. While networks could be trained from scratch, transfer learning is a common approach to reach high accuracy with less training cycles. Here, we start with the pre-trained model from ImageNet and then train the new model with the domain-specific annotated images. However, to the best of the author's knowledge, no-one has trained a visual classifier specifically for destination image measurement. In the next chapter, we introduce the author's implementation and evaluation of such a system.

## 3 Implementation of a Visual Classifier for Destination Image

Given the lack of an accurate classifier for destination image from photography, we decided to implement our own. While the leading benchmark systems in (general) visual classification tend to be highly complex (hundreds of layers and ten of millions of parameters) they also need expensive, powerful hardware to run. In our case, we are considering a more specific task (narrow domain) and aim for a model which can be trained and used with more common computing resources (e.g., a laptop with one GPU, or in cloud services such as Google Colab). We develop in Python, using the Keras open-source library for developing deep learning networks.

The standard workflow for a deep learning model is to prepare training data, build a pipeline to input the data, build a model, train the model with the data, test the model and then iterate on improving the model. To prepare training data, we need to decide on the set of visual categories for which the classifier will be trained then prepare a set of data for training which consists of tourism photography where each photo is annotated as belonging to one of those visual categories (every photo is considered as belonging to a single category. Multi-label classification is a subject for future work). Finally, train a model with the data and measure the accuracy on the test data set (we follow a standard convention of taking 20% of the data for testing).

To choose the visual categories for the classifier, we want to use the commonly accepted cognitive attributes of destination image as defined in the research literature. However, there is no single, widely accepted definition of destination image attributes. We start with the factors influencing destination image defined by [4] as the authors aimed for a comprehensive aggregation of all attributes that are considered in a destination. [17] wanted to measure destination image from Instagram photography (through a survey) and took a subset of the list in [4] which was best suited to cognitive identification (tangibly visible in an image). This led to a new list of 53 attributes where some attributes were more specific than [4] (e.g., "Flora and Fauna" became "Plants & Flowers", "Animals" and "Trees"). In a further refinement step, the author took this list of 53 attributes and cleaned it to 18 visually distinct categories based on aggregations (e.g., "Concerts" and "Cinema/Theatre" could be grouped into "Entertainment") and filtering out categories that are difficult to generalise for classification (e.g., "Arts and Crafts" and "Traditions" would look very different from place to place) (Table 1):

Table 1. Summary of [4]'s destination image attributes following aggregation and filtering by the author into visually distinct categories

To create the training data, we have noted that available classifiers have used public large scale annotated photo datasets such as ImageNet. However, these datasets are not annotated with tourism photography and destination image in mind and tend towards generic object detection tasks such as identifying a "car", "dog" or "football". Therefore, we found it important to source a new image training dataset specifically for our task. There is no golden rule for how many images are needed for training each category but following the TensorFlow tutorial for visual classification<sup>6</sup> where 3670 photos were used to train for 5 classes i.e., 734 photos per class, we also aimed to average at several hundred photos for each category. To find photos suitably representative of tourism photography in each category, we used Google Images search with the conjunctive query "tourism AND (label)" where label is the name of the visual category (categories like "plants & flowers" were split into two queries). The Fatkun Batch Bild plugin was used to download 500–800 images per category and the result was manually filtered to colour photographs without any overlays, leading to 100–500 photos per category for the training dataset. The final training dataset has 4949 photos, or an average of 275 photos per category. After training, the photos were discarded.

The initial architecture is a Convolutional Neural Network (CNN) with three convolution blocks, a max pooling layer in each, a fully connected layer on top with 128 units activated by a relu activation function and, in our case another fully connected layer with 18 units which outputs the result (a confidence value for each of the 18 visual categories). While the initial CNN had an accuracy score of 63–68% on the test dataset (after 20 epochs), we decided to explore the use of transfer learning to train our new model on top of the visual classification already learnt by an existing, re-usable model. Several models are available in Keras for transfer learning and after experimenting with results from a few options, we chose InceptionResNetV2 which reports

<sup>6</sup> https://www.tensorflow.org/tutorials/images/classification.

the best results on the ImageNet validation dataset yet is not the most complex to train with (56 million parameters)<sup>7</sup> . Further experimentation with the settings for the model training led us to choose max pooling and making all layers trainable. We loaded InceptionResNetV2 without the last two convolutional layers and added three convolution blocks with a max pooling layer in each, as in our original CNN. We run this new model for 10 epochs, and this model scored 90% accuracy on the test dataset.

Fig. 1. Confusion matrix for classification of the test dataset

We produced a confusion matrix to check if the accuracy is consistent across the categories (Fig. 1. NB the 18 visual categories can be seen on the vertical axis in the order they are classified). In the test dataset (983 images), 29 out of 34 images (85%) labelled building\_modern and 19 out of 24 images (79%) labelled monument were predicted correctly with the main confusion being 4 monument images labelled as building\_modern (out of 24). Other confusions visible in the matrix are desert images labelled as beach (5), landscape as water (4) and water as landscape (7), all of which are understandable given the common-sense similarities (deserts and beaches both have sand, landscapes also contain water). Overall, the classifier has correctly labelled the vast majority of images and has been consistent across categories.

<sup>7</sup> InceptionResNetV2 Simple Introduction, by Zahra Elhamraoui, Medium, May 16, 2020. https:// medium.com/@zahraelhamraoui1997/inceptionresnetv2-simple-introduction-9a2000edcdb6 (last accessed August 31, 2021).

### 4 Experiment

Having made available a visual classification model that can annotate photography according to a shortlist of destination image cognitive attributes with high accuracy, we will use the classifier to compare the perceived and projected destination image of a number of top destinations according to the Instagram photos posted by users and by the DMO respectively. Based on lists of most Instagrammed locations, we chose the cities of Paris, Barcelona and New York as well as the country of the Maldives, which also complement one another in the sense of each being quite distinct as a destination (Paris: romantic, historical; Barcelona: beach, modernism; New York: urban, entertainment; Maldives: water, relaxation). We use a Python library to download Instagram photos for each destination (i) from its official DMO account and (ii) according to the DMO recommended travel hashtag (this helps focus our data collection of tourismrelated photography). We discard the photos once we have classified them. Table 2 shows the selected account and hashtag and the number of photos acquired for each (the downloading took place in the second half of August 2021):

Table 2. Photos used in the experiment by destination

The classifier labels each individual photo with a set of 18 confidence scores (on a scale of 0 to 1), one for each visual category (in the order seen in Fig. 1). We accept one label per photo in that we select the visual category with the highest confidence score. Given an input of multiple photos, we can produce an array of 18 integers where each integer is the sum of photos labelled with a respective visual category. Vectors are representations of objects in multi-dimensional space. Following the use of vectors to represent data items in embedding layers of neural networks, where the items can be subsequently compared and learnt about (e.g., similar items are closer together and can be clustered), we will interpret the destination image array produced by the visual classifier as an 18-dimensional vector. Such a vector can be understood as a set of feature weights where the features are the visual categories and the weights are the presence of the feature in the dataset (to allow for comparison, all features need to be at the same scale so we take the no. of photos labelled with the feature divided by no. photos in the dataset to produce for every feature a value between 0 and 1). For example, the vector produced for the projected destination image (the DMO account) of Maldives is: [0.01663366 0.00871287 0.3829703 0.00435644 0.02336634 0.02336634 0.02930693 0.02534653 0.02336634 0.01029703 0.0150495 0.00475248 0.03960396 0.01584158 0.01346535 0.01029703 0.00990099 0.34336634]. The two dominant features are clearly beach (in the 3rd position with a value of 0.383) and water (in the last position with a value of 0.343) as all other features (all other destination image attributes) are not larger than 0.04. We can say that the DMO marketing of the Maldives (projected image) is heavily based on the beach and water attributes. We can use the cosine similarity of two vectors to determine how close the projected image (DMO) is to the perceived image (UGC) of the destination. Cosine similarity measures similarity in terms of the orientation of the vectors rather than the magnitude, which makes sense for the representation of destination image in n-dimensional space since each weight represents a feature of the destination and the overall image is determined by the comparative relationship of the feature values (e.g., beach and water is stronger than everything else) rather than the absolute values themselves. The resulting cosine similarity measure between the Maldives' projected and perceived destination images is 0.983 (cf. Table 3), indicating no significant differences. It seems Maldives visitors equally post in their majority beach and water photos. We can consider the Maldives case a validation of our approach with the visual classifier, as the Maldives as a destination is arguably limited to the two visual categories of beach and water (it seems neither accommodation nor gastronomy form a focus of destination photography there) and therefore it could be expected that these two features dominate both the projected and perceived images of the Maldives. How about cities like New York, Paris or Barcelona, which contain many different features that could be subjects of photography? Do the DMOs present a different image of the city than the visitors themselves? Table 3 shows the cosine similarity for each destination image, i.e., the closeness of the vectors of the destinations projected destination image (DMO account) and perceived

Table 3. Cosine similarity between vectors for the projected and perceived destination image of each destination

destination image (UGC).

There is a close similarity in all cases, suggesting the projected and perceived destination images are well aligned. On one hand, DMOs can achieve this by reusing UGC in their own destination marketing, a common approach today on platforms like Instagram. On the other hand, it suggests that Instagram users do share the same perception of the destination as the one being promoted by the destination marketers – a possible proof of the notion of the hermeneutic circle of representation [23]. This is the idea that we replicate and reinforce the media depictions we already know (in Paris, we have to photograph the Eiffel Tower, etc.). [10] considered that tourist photography may either reflect or inform a destination image since, consciously or unconsciously, tourists look for scenes that replicate their existing perceptions. While this result is positive for the DMO, there may some differences in the images which we can explore by looking at specific features (visual categories) in the destination image vector. Figure 2 shows comparisons of the city destination images along three features that all

Fig. 2. 3D plot of the destination images of the cities along three features (DI = perceived image, DMO = projected image)

of the cities are known for: gastronomy, historical buildings and modern buildings.

While the projected DMO imagery of Paris is closest to the perceived image in UGC (both points almost overlap), Instagrammers focus more on all three features than the DMO in New York (it may be the DMO is actively working on promoting other features of NYC) whereas in Barcelona the DMO tends to feature more gastronomy than the perceived destination image (which more strongly features both historical and modern buildings in the city). In particular, the New York DMO might be advised to focus more on gastronomy and historical buildings in NYC as this is certainly of more interest to Instagrammers, and the Barcelona DMO should consider that Instagrammers are attracted to both the historical and modern architecture of the city more than is reflected in their destination marketing.

### 5 Conclusions