*Article* **Reading Food Experiences from the Face: Effects of Familiarity and Branding of Soy Sauce on Facial Expressions and Video-Based RPPG Heart Rate**

**Rene A. de Wijk 1,\*, Shota Ushiama 1,2, Meeke Ummels 1 , Patrick Zimmerman 3 , Daisuke Kaneko 2 and Monique H. Vingerhoeds 1**

**Abstract:** Food experiences are not only driven by the food's intrinsic properties, such as its taste, texture, and aroma, but also by extrinsic properties such as visual brand information and the consumers' previous experiences with the foods. Recent developments in automated facial expression analysis and heart rate detection based on skin color changes (remote photoplethysmography or RPPG) allow for the monitoring of food experiences based on video images of the face. RPPG offers the possibility of large-scale non-laboratory and web-based testing of food products. In this study, results from the video-based analysis were compared to the more conventional tests (scores of valence and arousal using Emojis and photoplethysmography heart rate (PPG)). Forty participants with varying degrees of familiarity with soy sauce were presented with samples of rice and three commercial soy sauces with and without brand information. The results showed that (1) liking and arousal were affected primarily by the specific tastes, but not by branding and familiarity. In contrast, facial expressions were affected by branding and familiarity, and to a lesser degree by specific tastes. (2) RPPG heart rate and PPG both showed effects of branding and familiarity. However, RPPG heart rate needs further development because it underestimated the heart rate compared to PPG and was less sensitive to changes over time and with activity (viewing of brand information and tasting). In conclusion, this study suggests that recording of facial expressions and heart rates may no longer be limited to laboratories but can be done remotely using video images, which offers opportunities for large-scale testing in consumer science.

**Keywords:** RPPG and PPG heart rate; facial expressions; emojis; branding; familiarity; soy sauce


**2.1 Host immune response**

The cell surface receptors on *C. albicans* initiate adhesive interactions and invade the host cell using a series of proteins including adhesins and invasins. These immunodominant factors would trigger and stimulate a complex interplay of natural and adaptive immunity, posing interesting immunological response to the host. Cell-mediated (T cells) and innate immunity (macrophages, neutrophils and natural killer cells) are considered to be the most important line of defense against candidiasis [4] as they are recruited into the site of infection to exert protective effects. These include phagocytosis and antigen presentation, opsonization and production of chemicals for effective killing of the microbial cells. It must be emphasized therefore, that these responses comprise of different arms of the immune system (innate, cell-mediated and antibody-mediated) as shown in **Figure 1**.

### *2.1.1 Innate immunity*

White blood cells are produced and derived from multi-potent cells in the bone marrow known as hematopoietic stem cells and are found throughout the body, including the blood and lymphatic system [8]. Five individual types of white blood cells namely neutrophil, monocytes, lymphocytes, basophils, eosinophils [9] are involved in sustaining immunity [10].

Innate immune response is the dominant protective mechanism against disseminated candidiasis [11] and host defense against fungal infection depends on elimination of the fungi by phagocytic cells of the innate immune system, especially neutrophils and macrophages [12] at the initiation of infection before other immune cells are mobilized. Therefore, white blood cells are used to assess the working condition of body's immune system, to determine an active or chronic infection, identify the type of infection and also point to an allergic response or inflammation in the body [9].

Hence, quantitative and qualitative abnormalities of these immune cells are indications to different physiological conditions and particularly neutrophils and monocytes are associated with systemic candidiasis.