Principal stress is a vector, which can reflect the load transfer mechanism. Brandmaier et al. introduced the effect of the principal stress direction: when the orientation of bars is along principal stress directions in the frame, each bar can suffer the minimum shear stress so as for high stiffness structure [16]. The distribution density of principal stress trajectories reflects the stress intensity and the direction of principal stress represents the effect direction of principal stress. The trajectories guide the generation of truss-based structure.

By finite analysis method, three principal vectors at each node are solved as follows:

$$\begin{bmatrix} \sigma\_{\mathbf{x}} - \lambda\_1 & \tau\_{\mathbf{y}\mathbf{x}} & \tau\_{\mathbf{C}\mathbf{y}}\\ \tau\_{\mathbf{xy}} & \sigma\_{\mathbf{y}} - \lambda\_2 & \tau\_{\mathbf{C}\mathbf{y}}\\ \tau\_{\mathbf{x}\mathbf{z}} & \tau\_{\mathbf{y}\mathbf{z}} & \sigma\_{\mathbf{z}} - \lambda\_3 \end{bmatrix} \times \begin{split} \mathbf{r}\_j^{\text{principal}} = \mathbf{0} \times \mathbf{v}\_j^{\text{principal}} \neq \mathbf{0} \quad j = 1, 2, 3 \end{split} \tag{4}$$

The σ and τ are the normal stress and shear stress at each node, the eigenvector in three directions is donated as *v principal j* .

The principal stress trajectory starts from a seed point *Pseed* on the domain, the seed point move within the stress field until the point is out of the domain. In each moving, the point will find the nearest FEA node then move along with the principal stress direction that approximates to the pervious moving direction, as below:

$$P\_{l+1} = P\_l + \Delta \cdot \nu\_j^{princlial} \tag{5}$$

*Pi* is current position of the agent, is a constant, denoted to a moving step. *v principal j* is the principal vector of the node nearest to the *Pi*.

The agent by principal stress field is following:

Here, it uses Quad Remesh, a Grasshopper component, to divide the shell of sports helmets into quadrilateral panels under the guidance of principal stress trajectories. Finally, each panel is transformed into an infilling cell by Pufferfish, an open-source plugin of Grasshopper™, as illustrated in Fig. 8.

**Fig. 8.** Principal stress driven lattice infilling

# **3 Physical Experiment and Discussion**

The abovementioned wireframes are transferred into solid model by Dendro, an opensource plugin for Grasshopper™. All optimized helmet samples are fabricated by Stereo lithography Appearance (SLA) on FormLab3 and conducted compressive tests on SanSiZongHeng universal testing machines to verify the advantages of the two methods.

**Fig. 9.** Compressive testing

The experiment aims to demonstrate whether conformal gradient lattice infilling can improve the stiffness of sports helmets, as shown in Fig. 9. The lattice sport helmet only considering geometric conformal infilling is regarded as a comparative group. A just geometric conformal helmet is generated directly by Quad Remesh component without the guidance of any curves. The displacement of 1 mm/min is imposed on all the samples and the Force-Distance curves are shown in Fig. 10.

**Fig. 10.** The force-displacement curves of all samples

Compressive tests were performed by the proposed method and the results were illustrated as shown in Table 2. The weight of all the samples was controlled at 8.00 g. A Force-Displacement curve is obtained by three points bending experiments. According to the results of experiments, the linear stiffnesses of all simples are calculated with following Eq. (6):

$$k = \frac{\Delta F}{\Delta \delta} \tag{6}$$

where, *k* is denoted by linear stiffness. *F* is a force suffered by samples, and δ is a displacement produced by the force. *F*/δ is the slope of the force-displacement curve when the linear elastic deformation happens. The displacements at 0.5–0.6 mm and their responding forces are used for calculating the linear stiffness of samples, as illustrated in Table 2. Each structure produced five samples for comparison.

By comparison, it was found that samples vM-Voronoi, vM-Delaunay, and PS-Edge increase linear stiffness by over 0.6 times over geometric conformal lattice structures. This demonstrates that gradient lattice structure benefits stiffness optimization. Theoretically, the principal stress-driven method can present higher mechanical performance than the von Mises stress-driven method. Because the von Mises-driven method prefers to solve local stress concentration and achieves a gradient distribution of lattice material, whereas principal stress trajectories take the load transfer mechanism into account and regulate both lattice density and orientation. However, the results show that the von Mises stress-driven method has the highest linear stiffness among all samples. The main reason for this is that it adds an exponential factor to reinforce the effect range of the von Mises stress field, and then make the density distribution of lattice material reasonable, as referred to (15).

Another important consideration when infilling the lattice material is its anisotropy, especially when using the principal stress-driven method. In Group 1 and Group 2, two types of lattice structures were used: PS-Edge and GC-Edge, which are edge-cubic lattices, and PS-BCC and GC-BCC, which are body center cubic lattices. The edgecubic lattice is 90° orthotropic, which allows stress transfer direction to align with the material's maximum Young's modulus, resulting in high stiffness. However, the body

**Table 2.** Samples by additive manufacturing

center cubic lattice has a 45° orthotropic, meaning that the lattice mainly suffers from bending forces, resulting in low linear stiffness.

# **4 Conclusion**

This research presents two stress-driven MAS that release the full potential of lattice material and optimize the mechanical performance of sports helmets. By dividing the helmets into cells and infilling them with lattice material in a conformal manner under von Mises stress and principal stress, the density and orientation of the lattice material can be regulated to achieve functional, lightweight, and customized designs.

Compressive testing verified that the stress-driven methods produced products with superior performance compared to designs that only considered geometric conformal design. Additionally, the methods enable products with both high performance and aesthetically pleasing natural-like structures.

The MAS driven by the von Mises stress field allows for gradient material population, while also considering the minimum size features of the lattice material for successful fabrication. The MAS driven by the principal stress field can effectively regulate the populating orientation of the lattice material for improving stiffness. These design concepts can be further integrated with a lattice library to meet diverse functional requirements and personalized user preferences.

The proposed methods have the potential for broad industrial applications, such as in Unmanned Aerial Vehicles (UAVs) and automobiles. The methods can be integrated into automatic and iterative design progress, such as genetic algorithms, to yield diverse design outputs with robust mechanical performance.

**Acknowledgement.** This work was supported by the Research Development Fund of Xi'an Jiaotong-Liverpool University [RDF-17–02-44, RDF-SP-122], Industry Research & Development Projects [RP0028, RP0029] and XJTLU AI University Research Centre.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Simulation and Optimization**

# **Optimizing for Orientation in Complex Spaces**

Xuexin Duan(B) and Patrik Schumacher

Tongji University/Zaha Hadid Architects, Shanghai, China<EMAIL_ADDRESS>
**Abstract.** In response to the increasing demand for collaboration and knowledge exchange within Postfordist network society, both virtual and physical spaces are becoming more and more complex. Therefore the orientation within these increasingly complex and information-rich scenes becomes a problem that architectural design must address. The goal of this research is to upgrade architectural design competency in this respect by setting up a workflow for evaluating and optimizing the legibility of complex scenes. This paper introduces a novel research approach focused on the recognizability of salient interaction offerings within complex spatial settings, by using machine learning. A systematic workflow is being developed for simulations that appraise and rank design proposals with respect to the tradeoff between scene complexity and legibility. The authors explore the research through a series of simulation experiments concerned with semantic segmentation, i.e. with distinguishing and classifying relevant features in a large complex visual field. The paper first describes the method of setting up the measurement of complexity and ease of recognition, and then illustrates how a trained neural network can be used to evaluate and rank a series of design proposals (with systematically varied degree of complexity) on the basis of their recognizability. While the paper found that the hypothesis of a statistical inverse correlation or trade-off between complexity and recognizability holds, for each degree of complexity there are several design options with different degrees of recognizability. Therefore this approach allows to optimize the design of complex scenes in terms of the crucial criterion of legibility.

**Keywords:** Orientation · Legibility · Complexity · Spatial cognition · Machine learning · Design optimization

# **1 Introduction**

In response to the increasing demand for collaboration and knowledge exchange within Post Fordist network society, spaces are becoming more and more complex. Therefore the orientation within these increasingly complex and information-rich scenes becomes a problem that architectural design must address. This poses a serious challenge to human spatial perception. Research in spatial cognition, based on observation, involves a variety of behavioural measures. Beyond mere observation and performance measurement spatial cognition theory puts forward hypotheses about how people construct internal representations of space. [5, 12, 13]. The construction of internal representations becomes increasingly challenging as scene complexity increases.

As Friedman et al. [6] have argued, it is important to consider the legibility of an environment as a potentially important criterion when evaluating a post-occupancy environment. Many studies have shown that the complexity of a building plan is one of the most important factors affecting orientation in space [18, 15, 19]. However, people's spatial perceptions sometimes do not match the results from the analysis of the 2D plan. Prior research, e.g. Weisman [18] and Passini [16], had limited the study of the relationship between 'plan configuration' and navigation to corridor-type spaces. Their approach is difficult to implement in a more three-dimensional space, such as a complex atrium space. This paper extends the study of the relationship between legibility and architectural plan for atrium spaces. Our investigation of legibility is based on the first person perspective of situated human perception.

Spatial understanding can be partitioned into discrete components instead of being integrated into a single map [11, 2] One of the components is to locate and orient yourself in a space, which requires recognition of specific spatial features. In recent years, Deep Learning-based Convolutional Neural Networks have shown that trained models are able to recognize objects with human-level accuracy and speed [9, 10]. Recent research in deep-learning based semantic segmentation systems has focused on autonomous driving, industrial inspection, and medical imaging analysis [14, 7]. Architectural design has also drawn considerable attention to deep-learning-based semantic segmentation methods. For example, classifying furniture pictures based on design features [8], and comparing the visual similarity of interior design elements [4]. However, research in this area has focused on demonstrating the capability of successfully recognizing designs in the interior design field. And this study shows the potential of utilizing this technique to recognize salient interaction offerings within complex spatial settings.

The authors explore the research through a series of simulation experiments. We start with a method for measuring complexity. Then we present a method to appraise ease of orientation, and illustrate how a trained neural network can be used to do this, and thereby allowing us to evaluate and rank a series of design proposals.

# **2 Methodology**

The goal of this research is to upgrade architectural design competency by setting up a workflow for evaluating and optimizing the legibility of complex scenes. First the authors set up the formula for calculating the variety value which represents the diversity (complexity) of the destination in space. Then, using a deep-learning-based approach, the authors set up the evaluation model for ease of orientation, which appraises the level of legibility of a given space. The orientation evaluation includes three sub-measurement values: visibility, learnability, and recognizability.

#### **2.1 The Quantitative Definition of Variety**

Various research studies have demonstrated that the complexity of the floor plan influences the ease with which users can navigate within a building [15, 18, 16].This paper interprets complexity as diversity of spatial situations. In retail space, variety means many different kinds of units instead of mere repetition. Based on the K-means clustering algorithm, the measurement algorithm extracts four geometric attributes from the unit floor plan as input data: area, perimeter, shape proxy "width", shape proxy "length". Afterwards, these 4 numbers are combined to create clusters. The algorithm outputs how many types of the units can be clustered. Lastly, it will normalize the number of types as a value between 0 and 1. For instance, a floor plan with 10 units where all units are the same will result in 1 unit type, and variety value 0, i.e. this space is very simple. And when clustered into ten types, its variety value will increase to 1, which indicates a very complex space. Based on this we measure the degree of complexity of a space.

#### **2.2 The Evaluation of Ease of Orientation**

Our comparative appraisal of legibility or ease of orientation includes three aspects, namely visibility (a mere geometric precondition), learnability, and recognizability.

#### **2.2.1 Visibility**

For defining the visibility value, the authors utilize the Unity Perception package and the Unity Dataset Insights package. The workflow is as follows: First, we systematically build up an array of complex scenes. We then test/compare the scenes as follows: Step 1, place random targets. Targets represent attractions in the space, such as shops, kiosks, etc. Step 2: Place the camera viewpoints randomly within the space at eye level, so that all possible human perceptions can be sampled. Step 3: Assign a color code to all the target objects. As the simulation runs, each frame generates one view image and one labeled image. The data includes labeling information, color coding, and pixel numbers. Second, load the data and run the script. It calculates the total pixels of all the targets automatically using a Python script. So the higher the total pixel count of the targets, the more targets will appear in people's cone of vision, which means here the higher the space's visibility value. Third, normalize the data to yield a value between 0 and 1 that represents the comparative level of visibility. This is a simple, objective measure.

#### **2.2.2 Learnability**

As mentioned above, deep learning as part of machine learning algorithms has made significant progress in recent years in improving the accuracy of image semantic segmentation. In this research, the authors developed a new estimation method by constructing and training a deep learning network for semantic segmentation of architectural interior images. The authors use this first to derive a learnability value that simulates how fast a human can recognize the features of a new environment.

The measurement process includes two main parts. First, training data set generation.  Instead of manually segmenting each image, we developed a custom script that uses the Unity Perception Package to automate the process. After labeling and attaching specific color coding to the main architectural elements in the 3D scene: the escalator in magenta color, the balustrade in cyan, the glazing in dark blue, etc., we can start running the simulation. In the simulation, viewpoints are generated randomly in the scene to generate different view images and semantic segmentation images associated with the view. Second, training the network. This step involves loading the view images and semantic segmentations from the previous step to the learning machine, and the goal is to find out how much time the machine needs to learn to achieve a certain accuracy in its recognition/classification of features (Figs. 1 and 2).

**Fig. 1.** View image and Semantic Segmentation image of the view.

**Fig. 2.** Semantic segmentation image output for different views

# **2.2.3 Recognizability**

Here, in contrast to comparing how fast a machine can learn and improve to discriminate the architectural elements in different designed spaces, recognizability appraises how well a computer can recognize the architectural elements after it has completed its learning cycle. As previously stated, deep convolutional neural networks are the most powerful method for training a neural network model to extract relevant parts of an image's features. However, how to combine techniques with specific domain knowledge and how to collect data has become a critical issue. To compare how well a trained machine system can recognize specified features in different designed interior spaces, the commonly used deep learning algorithm YOLO (You Only Look Once) was used because of its fast speed and accuracy [17].

In this research, we focus on the escalators as the most significant elements of interior atrium spaces. In our approach, we trained the YOLO model to detect a new custom object (the escalator) based on the trained COCO dataset (Common Objects in Context is a dataset provided by the Microsoft team that can be used for image recognition) instead of starting from scratch. Prior to deep learning training, data annotation is required. By using the unity perception package, the authors can automatically crop the region and label each image to generate a large amount of training data quickly, rather than manually labeling each image in a common way. There are 600 images prepared, 80% of the total collected data is used for training, and 20% is used for validation.

Instead of using training time data as in the learnability simulation, the detection model solves for identifying (recognizing) the escalators in the view image and in addition outputs a confidence value for each identified element as to how confident the network is in classifying the element as an escalator. The average of all the confidence values generated in a scene feeds into the overall performance measure for this scene, indicating the scene's recognisability (Fig. 3).

**Fig. 3.** Example results from detection model, in image 1 the machine detected two escalators with confidence value of 0.86 and 0.89, the accuracy value for this view image 1 is 0.875.

# **3 Prototype Development and Trade-Off Graph**

#### **3.1 Atrium Generator Prototype Setup**

The retail atrium was chosen as spatial type capable of exemplifying the problem to be investigated, namely to maintain legibility in the face of complexity. We developed a parametric atrium generator to create a set of systematically differentiated sample spaces. The atrium designs vary the void shape and the depth of the slabs up to the glazing line. The atria are radially divided into 12 segments. In the simplest case these 12 segments are equal. In the most complex case they are all unique.

The value of variety increases with the total number of different segment types. At first the atria operated across two floors above ground, later three floors. A sequence of different design iterations was generated. This sequence measured variety values ranging from 0 to 0.91. In principle, there are unlimited numbers of ways to generate designs with the same variety value. The authors selected three design iterations for each variety value. In total there are 34 design iterations with 12 different variety values (for variety = 0, there is only one design iteration as all the units are the same). For example, in Fig. 4, the variety value increases from left to right, which means the space is becoming more complex in terms of the diversity criteria.

**Fig. 4.** Plan diagram spectrum for one floor showing the how the variety value increases while the spatial scene becomes more complex.

#### **3.2 Trade-Off Graph**

The authors hypothesized that increasing the complexity will reduce the legibility of the space. Indeed, we observed this inverse relationship between complexity and legibility. We see this as a trade-off, since both complexity (delivering diversity of destinations) and legibility (affording ease of orientation) are desirable.

After generating the design iterations with a wide spectrum of variety values, we conducted our learnability and recognizability tests for each of the designs. By mapping all the values for variety first against the obtained learnability and recognizability scores, we get the maps in Fig. 6. Based on statistical theory, when the R2 value is bigger than 0.6, this means the two parameters have a strong correlation. We also can see from the mappings that for each of the variety values, despite the overall correlation holding, there are several designs with different legibility (learnability/recognizability) scores, i.e. for each visibility value there are relatively better performing designs we might select, i.e. a design with both high complexity and relatively high legibility.

#### **3.2.1 Learnability Trade-off Graph**

With the same 34 design iterations spread over 12 design variation values, we executed the learnability experiment to measure how long it takes for the machine to achieve 95% accuracy. For instance, when the variety value = 0. It only took 45s for the machine to learn to recognize the architectural elements and reach an accuracy of 0.95. In comparison, a design with high variety (variety = 0.91), requires 134s of learning to reach the accuracy of 0.95. After training for all the design iterations, we obtained the graph in Fig. 5a which also shows that there is a strong correlation between variety(X axis) and learnability (Y axix). The R2 value is 0.7461, i.e. significantly above 0.6.

#### **3.2.2 Recognizability Trade-off Graph**

Instead of comparing the time taken to reach a certain accuracy between different design iterations, the recognition experiment uses the same trained neural network to test the recognition accuracy with different design iterations. For the same series of designs, 50 images were chosen from each of the 12 design variety brackets. In total, there were 600 images constituting the training dataset. After training, the neural network can achieve an accuracy above 0.8 for the validation set.

After building the neural network, the next step was to test the recognition performance of the neural network for the different design options with different degrees of variety. To test the different designs, 10 new perspectival view images for each design have been fed to the neural network. We then obtained the average accuracy figure for each design, which for us represents the recognizability value of this design. For instance, when the design variety equals 0.00, the average accuracy is 0.84 for the 10 images, and when the design variety equals 0.91, the average accuracy dropped to 0.66. After we run the recognition test for all the options, we get the graph shown in Fig. 5b. We can see from the output figures that the distribution of accuracy correlates with the variety value. This indicates that the recognizability value has a strong correlation with design complexity.

**Fig. 5.** Variety-Learnability (**a**), Variety-Recognizability (**b**) trade-off graphs

Both learnability and recognizability were explored via neural networks trained to semantically segment images of a complex scene and classify features like escalators. According to these experiments, there is a strong inverse correlation between variety and legibility, appraised in its aspects of learnability and recognizability. Since for each variety measure we can generate and test multiple designs, we can use this process to optimize the trade-off between variety and legibility.

#### **4 Further Design Iterations**

#### **4.1 Learnability Experiment**

It has already been mentioned that there are an infinite number of ways to generate a design for a particular variety value. Our goal is to evaluate and optimize in order to find the most legible solution. The authors compared three design options with similar plan configurations and the same variety value of 0.62, but with changing the slab edge curvature. Option A has a large curvature on the slab edge, Option B has a faceted slab edge, and Option C has a gentle curve on the slab edge. According to the learnability tests, option C exhibits a high learnability value. This is in line with our intuitive expectation that the smooth version will perform better than the faceted version (because its generates less visual noise).

**Fig. 6.** Three design iterations' legibility comparison. In option C, the visibility value is 0.745, while the other two are 0.713 and 0.736, and in option C, the training time is 3:45 while Option B's training time is much longer as it is 5:23, and Option A is fairly close as it is 3:54.

#### **4.2 Recognizability Experiments**

In addition to the prototype experiments, the authors also examined the methods on a real design project. It is a 5-floor atrium space in a mixed-use (retail and cultural) complex. Using two design iterations as examples, we applied our recognisability appraisal method. When all the necessary functional program plan layouts have been met, it is typically depending on the architect's intuitive qualitative appraisal to choose a number of design option that meet the programme criteria. In contrast to this subjective choice, our method allows for reproducible selection based on a quantitative process that can be critiqued and improved upon.

The evaluation and optimization process was applied as follows: first, we randomly selected 10 viewpoints to get 10 view images from each of the designs. Then we input these images to the neural network (trained on the proto-types) to get the escalator recognition performance with accuracy values for each image. Then we averaged all the accuracy values to get an overall recognizability value for the design option in question. Any mistaken detection is taken into account with its confidence value as negative value. Due to mistaken identifications both options' recognizability values were rather low, but this measured difference is significant. For option 1 it was 0.277 and for option 2 it was 0.023 (see Fig. 7). It was interesting that the network was performing at all in a space that was quite different from the training set. In any event, absolute values matter little. The purpose of our method is to get a ranking that guides selection. The result suggests option 1 is better in terms of recognizability. From our intuitive point of view, option 1 also looks more legible. This illustrates how a neural network can be used to evaluate and rank a series of design proposals based on their recognizability.

The evaluation process is quite efficient and easy to operate. You can upload either a screenshot or a rendering of the design space. As a result, a value of recognisability will be output for this space. Many questions and problems remain to be addressed. For instance, when feeding new images that substantially deviate from the original training

**Fig. 7.** in image 03 of option 1, the confidence value outputs are 0.76, 0.77, 0.91, but it missed some escalators, so the accuracy value is 0.407 (= (0.76 + 0.77 + 0.91 + 0 + 0 + 0)/6). And for option 2, it missed out on all the escalators, and also incorrectly "recognized" some balconies as escalators, so the accuracy value for this view image of option2 is –0.246 (= (–0.61–0.62 + 0 + 0 + 0)/5).

set, overall accuracy is not high. Hence, it will be necessary to expand the database with more interior images of atriums in the future in order to enhance the accuracy of the data. Second, the current neural network is only trained to recognize the escalator as a key element of the atrium. However, it is possible to extend it to incorporate other key architectural elements as well (Fig. 8).

**Fig. 8.** Example outputs from detection model tested on the photorealistic renderings.

# **5 Conclusion and Future Work**

The objective of this research is to provide a methodical way of posing the issue of legibility. Our work so far only maps out an initial sketch schematic for the operationalisation of the legibility criterion in its relation to complex scenes. The atrium typology is just one scenario among many. This method setup can be abstracted and generalized, and then tailored to support all urban, architectural and interior design processes. Both complexity (spatial diversity) and legibility (ease of orientation) are considered positive characteristics. While the hypothesis of an inverse correlation or trade-off between complexity and legibility holds, there are several design options with different degrees of legibility for each level of complexity. The methodology can thus help find the sweet spot in the trade-off between these two requirements, or else we can take a certain level of complexity as given, and then identify design options that maximise legibility without compromising complexity. In this way, the design of complex scenes can be optimized according to the crucial criterion of legibility.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **A Virtual Reality Window View Evaluation Tool for Shading Devices and Exterior Landscape Design**

Wentao Zeng(B) and Hanyi Zhang

College of Environmental Design, University of California, Berkeley, CA, USA<EMAIL_ADDRESS>
**Abstract.** The window view is an important part of the daylighting design. The current window view analysis based on daylighting metrics does not respond well to user preferences. This study uses an office with a courtyard in Berkeley, CA, USA, as a case study to create a virtual reality-based window view evaluation tool and workflow to analyze the impact of different types of shading devices and different levels of exterior landscapes on user perception. This tool combines quantitative data based on daylighting metrics and users' subjective and physical responses with qualitative analysis based on user feedback and preferences. A twoway ANOVA was conducted in the study to demonstrate that the independent and interactive impacts of shading devices and exterior landscapes on user perception and satisfaction. The results show that users prefer shading types that block less of window views even though they may cause a higher probability of glare. Besides, advanced landscapes tend to enhance user satisfaction with shading devices. This new window evaluation method will help architects make more comprehensive decisions in shading device type selection and exterior landscape design.

**Keywords:** Window view · Shading device · Exterior landscape · Virtual reality · User perception and preference

# **1 Introduction**

The study of daylighting is an important part of architectural design. Architects and designers often overlook the quality of the occupant's view, an essential analysis factor of daylighting design. The research indicated that reasonable window views can reduce human physical and mental discomfort [1]. Moreover, creating a connection between landscapes and humans by enhancing window views has been proven to positively impact the health of the occupants [2]. Thus, daylighting and window views are increasingly being integrated to consider the impact on occupant health and well-being. As a medium between the interior daylight environment and the exterior landscape, shading devices must balance daylight performance and window views. Appropriate shading devices have been shown to optimize multiple aspects of daylighting, such as daylight distribution, glare, and views [3]. The selections of shading devices and landscaping in the previous studies rarely considered end-user preferences and were mostly based on daylight simulation results and subjective design by architects. Thus, to design window views conducive to human health and sustainability, the selection of shading devices and the design of the landscape should accommodate the visual comfort and preferences of users while satisfying daylight standards.

Virtual reality (VR) as an excellent medium can combine simulation-based daylighting analysis with user-based immersive analytics to integrate daylighting metrics and visual preferences of users in the early stages of the shading device and landscape design. VR is becoming an effective alternative for the evaluation of interior visual environments because it has been proven to be superior to video and pictures for subjective perception, and it allows for controlling selected variables, analyzing causal relationships, and saving time and costs spent on real building measurements [4, 5]. Abd-Alhamid et al. [6] confirmed the importance of the information content seen in the window views by analyzing the observation data and feedback from users at different locations in the VR scene and showed that the design of the window view has significant implications for the health and well-being of building occupants. Chamilothori et al. [7] combined VR and wearable biometric devices to study the effect of different shading façades and scenarios for user vision. The results showed that the different patterns and geometries of the shading façade influenced the users' subjective visual evaluation and physiological responses. Lee et al. [8] introduced a method for evaluating view clarity through VR. The study revealed that the geometry and material of the shading system can affect the clarity of the exterior landscape to the degradation of the quality of the view. To sum up, VR has a strong potential to be used as a tool to connect user perception and architectural research. Therefore, more and more architects and designers have the opportunity to gather information about end-user requirements and preferences for window views through VR to make more informed decisions during the preliminary design stage.

The research focused on shading devices and exterior landscapes as the two main points of attention in the window views study. Previous analysis of window views is typically based on a relatively elaborated assessment of the LEED v4 Quality Views (QV) credit [9]. However, this evaluation metric does not require all view criteria to be met the credit, and therefore results in window views that are often low quality and do not respond well to real user feedback [10]. Moreover, other visual perception effects such as glare and thermal discomfort are not taken into account when studying the window views. At this time, there are no established methods to guide designers and researchers in investigating users' perceptions of window views. Based on this, this study proposes the following questions in different stages of workflow:

This study aims to develop a window view evaluation tool and workflow that combines subjective and objective analysis methods. Firstly, the method from daylight models to immersive virtual environments was studied, and the method of constructing a more comprehensive VR window view analysis tool was summarized. This new evaluation tool was then used in the design of shading devices and exterior landscapes for an office in Berkeley, California, and its application to window views perception and feedback for users were explored. This VR window view evaluation tool allows architects to combine daylighting metrics and user preferences at an early stage of design to comprehensively compare and select options for shading devices and landscapes. In addition, the VR window view evaluation tool proposed in this study has been assessed by usability and universal applicability and can be applied to other types of spatial analysis.

### **2 Method**

The primary goal of this study is to propose an innovative VR window view evaluation tool and workflow. Thus, the impact of shading devices and exterior landscapes in window views was studied through immersive virtual environments to help architects find a balance between daylight performance and window views. The study uses an office with a courtyard in Berkeley, CA, USA, as an example. Rhinoceros 3D and Grasshopper were used to create a model an office model with shading devices and exterior landscapes, which allowed defining types of shading devices and different levels of exterior landscapes as parametric variables. Daylight simulation and analysis are completed by embedding the required weather files and material data through ClimateStudio to select shading devices that meet daylight requirements. Moreover, the Radiance Render function of ClimateStudio creates scenes of 360 high dynamic range renderings (HDRR) with different shading devices and landscapes. The VR scenes were created by importing high dynamic range renderings into Unity and adding an interactive interface to complete the user evaluation system. Finally, daylighting metrics, user feedback, and physical responses to different VR scenes were compared and analyzed through case studies to help the architects select the appropriate shading devices and exterior landscapes for the office space. Figure 1 shows the research process and methods of this study.

#### **2.1 Experimental Model**

The research uses a south-facing office with a courtyard on the first floor of an office building in Berkeley, CA, USA as the case study. The office's length, width, and height are 5 m, 3 m, and 4 m respectively. Moreover, the office has an exterior courtyard without landscaping with a length and width of 15 m and 10 m. In addition, the office has only one window on the south wall, which is 2.9 m in length, 1.6 m in height, 1m in distance from the floor, and with a wall-to-window ratio of approximately 40%. The weather type in Berkeley is a Warm-summer Mediterranean climate (Csb in the Köppen climate classification) with long, mostly sunny summers. Thus, the architect needed to choose appropriate shading devices and exterior landscapes by analyzing window views to create a comfortable daylighting environment. Based on information from the site,

**Fig. 1.** The workflow specified for the research.

the office was 3D modeled by Rhinoceros 3D and Grasshopper for daylight simulation, system development, and window view analysis (Fig. 2).

**Fig. 2.** 3D modeling based on real environment: real office space (left), 3D model for case study (right).

Different types of shading devices and external landscapes were set as independent variables in this study to investigate the comprehensive effect of user responses in window views. The space size, wall-to-window ratio, and materials of the case model are fixed in the analysis. Figure 3 shows the three levels of exterior landscapes created in the study: a no landscape with brick-paved ground, a normal landscape with partial grass and shrubs, and an advanced landscape with more trees and artwork. Moreover, the study selected five types of shading devices in pre-experiments using Grasshopper and ClimateStudio to meet the requirements for spatial Daylight Autonomy (sDA), i.e., the interior space receives at least 30fc of daylight for at least 50% of the workday [11]. Six types of shading scenes including no shading, low-density horizontal louvers (LH Louvers), high-density horizontal louvers (HH Louvers), low-density vertical louvers (LV Louvers), high-density vertical louvers (HV Louvers), and special egg crate louvers (EC Louvers) were set up in the study to investigate the effect of shading type and density on the visual comfort of the users (Fig. 3).

**Fig. 3.** Three different landscape levels and six different shading devices for window view analysis.

#### **2.2 Daylight Model**

The daylight model integrates 3D models, weather data, and materials for window view simulation and rendering of office scenes with different shading devices and exterior landscapes. The EPW weather data files for Berkeley were imported into the daylight model through ClimateStudio using occupancy times from 8 a.m. to 6 p.m. The materials used for simulations and renderings of the office, shading devices, and landscapes are all from the ClimateStudio materials library (Table 1). Daylight metrics were used as supplementary information for window analysis to help architects and users understand the impact of different shading devices and exterior landscapes on interior daylight performance and glare protection. In the study, sDA was used to analyze the annual daylight performance of the space to evaluate different types of shading devices. The analysis grid used to calculate the sDA was located at a table height of 0.73 m, with the sensor points spacing was 0.6 m and their distance from the walls of 0.5 m (Fig. 4). Moreover, the study used Spatial Disturbing Glare (sDG) based on the Daylight Glare Probability (DGP) metric to analyze the annual average glare of the space, i.e., the percentage of space that experiences Disturbing or Intolerable Glare (DGP *>* 38%) for at least 5% of occupied hours. The analysis grid for the sDG metric used the same sensor spacing as the daylight analysis grids. The default view was located at 1.2 m off the floor (eye height for a seated observer) and the calculation was based on hourly DGP values for eight different view directions at each position in the space (Fig. 4). As a common assessment method of window views, QV credits in this study were always at 100 due to the area of the office (length less than 7.5 m). Thus, the window views analysis method using QV could not be used in this project to investigate the impact of different shading devices and exterior landscapes in more detail. Radiance Render of ClimateStudio was used in this study to simulate the 360 HDRR by daylight model used to create the virtual scene for the window views analysis. The time of the simulation for the scenes was chosen at the highest sDG value of the year (12.30 pm on December 21), and the rendering position P was 2 m from the window at a height of 1.2 m off the floor (Fig. 4).

**Table 1.** Optical properties for objects in the simulation.

**Fig. 4.** Example of sensor points with views used for sDA simulations (left), sDG simulations (center), and window view rendering (right).

#### **2.3 Generation of VR Window View Evaluation Tool**

The VR window view evaluation tool created immersive virtual scenes for window view assessment by embedding 18 different 360 HDRRs in Unity3D (including a combination of 3 levels of exterior landscapes and 6 types of shading devices in the experimental model). The tool had three main functions to assist users in better completing the window view evaluation. Firstly, a scene-switching function was created to help the user randomly change to the next scene after completing the evaluation of one scene (Fig. 5). Secondly, as glare is an important factor in window view evaluation, the system added a glare observation function (Fig. 5). The system showed the area where the glare existed in the VR scenes by the partial false color (pink area *>* 2000cd/m2). Compared to the full false color indication of glare, the partial false color only indicates the glare area to allow the user to intuitively understand the luminance situation in the scene and help the user to complete a more comprehensive window assessment. In addition, users can record their emotions and satisfaction with the different shading devices and exterior landscapes through the evaluation function after completing the observation of each scene (Fig. 5). As listed in Table 2, the evaluation function consisted of 6 questions, all of which are measured on a seven-point Likert scale (mostly 1 = fully disagree and 6 = fully agree). The data collected by the system helped the architects better understand end-user preferences and feedback.

**Fig. 5.** The three main functions of the VR window view evaluation tool: scene-switching function (left), glare observation function (center), and evaluation function (right).

**Table 2.** Evaluation questionnaire for window view study.

#### **2.4 Experimental Design**

Instead of traditional physical window view analysis, the study was observed and evaluated through the immersive virtual environment of the VR window view evaluation tool to analyze users' perceptions and preferences for different shading devices and exterior landscapes. The headset used for VR window view observation in the study was an Oculus Quest 2 with a field of view of 100°, a resolution of 1832 × 1920 pixels per eye, and a refresh rate of 90 Hz. The study created a VR environment through Unity3D that can be used for observation of Oculus devices and interaction with the system through controllers. Moreover, a total of 30 participants (15 males and 15 females) participated in this study. Participants were limited to a range of 20–32 years (mean age: 23.8, SD = 3 years). The study used a within-subject experiment design i.e. each participant was tested on the same 18 scenes to eliminate individual differences between participants. In addition, participants were asked to test individually in a real office space. After confirming familiarity with the equipment and experimental procedures, participants were exposed to neutral scenes and tested for basal heart rate in VR, and then a series of scenes were observed in random sequences. While observing the scene, participants were required to use the glare observation function to identify areas of glare in the window and to receive 30 s of heart rate monitoring (Fig. 6). At the end of each scene observation participants were asked to complete the evaluation questionnaire to collect participants' emotions and satisfaction with the shading devices and exterior landscapes in the window views. The average observation and assessment time for each scene is about 2 min. After completing a window view evaluation of a scene participants could switch to the next scene until they completed 18 scenes. Upon completion of the experiment, participants were requested to complete the feedback on window views and system usability. Furthermore, all participants provided written informed consent before the study and were compensated for their participation.

**Fig. 6.** Participants performed window view evaluation and monitored heart rate in VR scenes.

# **3 Results Analysis**

#### **3.1 Daylight Analysis**

Although the main focus of this study is on window view assessment, daylight analysis as a basis for window view analysis helps architects and users understand the daylighting performance and glare probability of interiors with different shading devices and exterior landscape conditions. As shown in the daylight simulation results in Fig. 7, the five different shading devices in the study kept the annual sDA at 55.0–72.5%. Among them, LH Louvers and LV Louvers had the best daylight performance, with an average annual sDA of 71.3%. Moreover, the five different shading devices reduced annual sDG by 19.4–38.8% compared to the No Shading scene (Fig. 7). HH Louvers demonstrated the best glare protection with an annual sDG of 0.6%. Furthermore, compared to No Landscape and Normal Landscape, Advanced Landscape decreased the DGP of different shading devices by an average of 11.4% in the scene on December 21 at 12.30 (Fig. 7). Daylight analysis proved that Advanced Landscape, which has tall trees and artwork, has an impact on glare protection for users. In addition, the daylighting metrics provided basic information for participants in the window view evaluation and additional support for the architects in the window view analysis.

**Fig. 7.** Daylighting metrics in different window view scenes.

#### **3.2 Window View Analysis**

The study analyzed 30 sets of scoring data and physical response data obtained from the VR window view evaluation tool by experiments. The Kolmogorov-Smirnov test showed that the data for all dependent variables were normally distributed, therefore a two-way ANOVA was used to determine the independent and interactive impacts of two factors, the level of exterior landscapes and shading types [12]. Statistical analyses of data were performed in Python, using the toolbox from Pandas, Matlibplot, and Statsmodels. For the multiple independent hypotheses, a Bonferroni-corrected significance level α of 0.0014 is used for the within-subject factor analyses. Thus, the level of the strong effect of significance is 0.001.

#### **3.2.1 Subjective Responses**

In this section, user perceptions and feedback regarding shading types and exterior landscape levels in window views were investigated. A two-way ANOVA was performed for each dependent variable to detect the effects and interactions of shading types and exterior landscape levels on users' emotions. The statistical analysis showed a significant effect of shading devices and exterior landscapes on perceived pleasure, interest, and excitement (SSlandscape\_pleasant = 611.411, Plandscape\_pleasant *<* 0.001, SSshading\_pleasant = 149.817, Pshading\_pleasant *<* 0.001; SSlandscape\_interesting = 549.081, Plandscape\_interesting *<* 0.001, SSshading\_interesting = 182.326, Pshading\_interesting *<* 0.001; SSlandscape\_exciting = 601.893, Plandscape\_exciting *<* 0.001, SSshading\_exciting = 180.542, Pshading\_exciting *<* 0.001). The results of the analysis supported the research variables that different shade types and levels of landscape affect participants' responses to the degree of pleasure, interest, and excitement of spatial perception (Table 3). The study demonstrated that participants probably were more pleasant, interested, and excited in advanced landscapes or types of shading devices that blocked less of window views (Fig. 8).

A two-way ANOVA was conducted on participant satisfaction data to investigate the independent and interactive impacts of independent variables on shading device satisfaction (SDS), glare satisfaction (GS), and exterior landscape satisfaction (ELS), (SSlandscape\_SDS = 140.011, Plandscape\_SDS *<* 0.001, SSshading\_SDS = 921.422, Pshading\_SDS *<* 0.001; SSlandscape\_GS = 47.004, Plandscape\_GS *<* 0.001, SSshading\_GS = 799.637, Pshading\_GS = 0.008 *>* 0.001; SSlandscape\_ELS = 629.559, Plandscape\_ELS *<* 0.001; SSshading\_ELS = 173.837, Pshading\_ELS *<* 0.001). The results of the study showed that the interaction of different levels of landscape and different types of shading devices was significant in terms of shading device satisfaction, but not in terms of glare impact and exterior landscape satisfaction (Table 4). Participants tended to score higher on shading device satisfaction and diminished scores on glare discomfort in more advanced landscapes (Fig. 9). Moreover, when the landscape was more obscured by some type of shading device, participants tended to give lower scores on this type of shading device (Fig. 9).

#### **3.2.2 Physiological Responses**

The study used a two-way ANOVA to analyze the independent and interactive effects of shading devices and external landscapes in the window views on physiological responses

**Fig. 8.** Interaction plot for average rating of pleasant level (left), Interesting level (center), and exciting level (right) was perceived in different scenes.

(Table 5). The results showed no significant interaction between shading types and landscape levels (Pshading\_landscape = 0.008 *>* 0.001). However, the effects of both shading devices and exterior landscapes on heart rate were significant. Figure 10 shows that the advanced landscape scenes had the lowest heart rate differences, meaning that participants were probably in the calmest emotions at that time. Moreover, for scenes with HV Louvers, excessive view blockage and glare may have contributed to the high heart rate differences of participants, meaning that participants may have had uncomfortable emotions at that time (Fig. 10).

#### **3.2.3 User Feedback**

Besides the scoring data of window views, the experiment also requested feedback from the participants through questionnaires. Questionnaires are an effective way to collect information on user preferences and behaviors. The study obtained qualitative data on user preferences and usability assessment for the VR windowing evaluation tool through user feedback (Fig. 11). Regarding the window views provided in the study, most users stated that "shading devices that block less of view give me a better visual experience", "I definitely prefer advanced landscapes, it makes me physically and mentally happy", and "I think the combination of low-density louvers and advanced landscapes is my preferred window views". However, a small number of participants indicated "I personally dislike excessive direct daylight, so I prefer shading devices with good shading, even if most of the view is blocked", and "I prefer normal landscapes with only grass and shrubs, such exterior landscapes make me peaceful". The analysis of user feedback showed that users are more concerned about getting relatively unobstructed window views than about glare. Therefore, LH Louvers and LV Louvers with more open window views are the preferred types of shading devices for users. Furthermore, the advanced landscape became the preferred level for most participants. User feedback helped architects to understand user preferences more intuitively and select appropriate shading devices and exterior landscapes.

User feedback also provided a system usability assessment for the study (Fig. 11). On the positive side, users commented that "it provides me with a good visual immersive experience", "the scenes are rendered realistically" and "the system is impressive, comfortable to observe and the interaction menus are responsive". However, users also reported that "I would like to have more interaction with things in the environment", "I wish I could move around in the scenes", and "some parts of the scenes are not clear due to overexposure". Participants' feedback on the system mostly focused on improving

**Fig. 9.** Interaction plot for average rating of satisfaction with shading devices (left), glare (center), and exterior landscapes (right) in different scenes.

**Table 5.** Results of two-way ANOVA testing the effects and interaction of landscape levels and shading types on heart rate.

**Fig. 10.** Interaction plot for average heart rate differences in different landscape levels and shading types.

hardware technology and enhancing interactive functions. The user feedback will inform and assist in the future development of the VR window view evaluation tool.

**Fig. 11.** Excerpts from questionnaires of user feedback.

# **4 Conclusion**

The evaluation of window views requires consideration of several aspects, including shading devices, exterior landscapes, wall-to-window ratios, window materials, observation locations, etc. This research focuses on shading devices and exterior landscapes in the window view evaluation. The VR window view evaluation tool and workflow developed in this study can help architects to comprehensively assess window views and apply them to the design of shading devices and exterior landscapes. The originality and value of this window view research are as follows:

The results of the window view analysis showed the effects of different shading devices and landscape levels on user emotion, satisfaction, and physical responses. Most users preferred shading devices that block less of the view, such as LH Louvers and LV Louvers, even if they have a higher DGP. Moreover, most users preferred the advanced landscape, which tends to increase user satisfaction with shading devices. Furthermore, the user feedback presented in the study can help users to select shading devices and landscape levels that have specific preferences. These analyses of user preferences helped architects to make more comprehensive window view evaluations and design decisions.

The scenes used in the VR window view evaluation tool for this study are simulated based on the case model, which can be replaced with different window view scenes through the design workflow proposed in the paper. Moreover, the system has only one fixed observation position that does not allow for a multiple-perspective window view observation. Therefore, the collection of user perceptions and feedback may be defective. A movable immersive observation environment and more comprehensive physical detection equipment may be added in further studies to complete a more comprehensive window view evaluation tool.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Virtual Reality Architecture Teaching Application Based on Unity Platform—Taking a Small Architect's Metaverse Application as an Example**

Yihan Lu, Zhouyu Zhang, Yaoping Zhang, and Li Li(B)

Southeast University, Xuanwu, Nanjing 210096, China<EMAIL_ADDRESS>
**Abstract.** Children's architectural education is mainly built by assembling existing prefabricated blocks. However, most of the existing architectural education projects focus on PC applications, which are important for transforming architecture into practical results. Taking the achievements of the VR architectural knowledge popularization education project as an example, this paper demonstrates the application of the combination of virtual reality technology and architectural education in the future architectural design education for children. The app adds a follow-up connection to reality that is missing from traditional VR experiences. First of all, this application built a virtual reality building on the unity3d platform, controlled vr equipment through the OpenXR library, designed a program to build a house with prefabricated modules in the application, and finally exported the virtual results as DXF files to the laser cutting equipment, and assembled them in reality. It retains the interest and diversity of the virtual meta-universe and can increase the authenticity of the combination of virtual and real life in the education links to improve hands-on ability. This application through the students in a three-dimensional virtual environment to experience the process of building, but it also can use laser printing technology to transform students' learning results into real results in order to realize the application of virtual reality technology in architectural education.

**Keywords:** Virtual reality · Architectural education · Block cutting and assembling · Construction logic

# **1 Introduce**

Building education for children is a common type of education. Most of the educational applications of VR are directly applied to architecture exhibitions or purely virtual world construction, lacking the process of hands-on practice, and the results have not been turned into reality. Based on relevant literature research results and user research, this paper introduces the logical framework of a VR building construction system and, on this basis, develops a set of VR building teaching application design frameworks and design strategies, which explores a path for the combination of virtual reality and children's architectural education in the future. Teaching for children aims to enhance the sense of experience, exercise practical ability, and enhance the understanding and interest in architectural knowledge. Explore a VR teaching system platform based on virtual reality technology to improve the diversity and sense of the experience of architectural education (Fig. 1).

**Fig. 1.** Application scenario screenshots

# **2 Background**

#### **2.1 Current Status of VR Education Development**

At present, elementary school architecture popular science education mainly stays at the display level of traditional multimedia. Two-dimensional image and audio-video display cannot effectively enhance the interaction between teachers and students, students' sense of participation is not high, and the degree of knowledge absorption and other aspects of learning performance is not good. The pattern between teachers and students is that one teacher corresponds to many students, and problems can not be solved in time, as well as how to save the architectural design, construction, and assembly games. The application of virtual reality technology in a wide range of fields in China, but the penetration rate of application in education is still very low. First, because of the high cost of virtual reality technology equipment and manpower; Second, public awareness is not enough. [1] For teenagers, the lure of new technology is very strong. Touchstone Research, an online marketing research company, surveyed 500 teenagers between the ages of 10 and 17 and released statistics on the awareness of virtual reality technology: 79% of people have heard of virtual reality technology, 68% of people have gradually learned about it, and 47% of people have some or a deep understanding of virtual reality technology. Therefore, the prospect of virtual reality technology among teenagers is still extremely optimistic. Many teenagers aged 10–17 are very interested in virtual reality. "VR virtual reality devices have a far greater impact on minors than TV, computer and other products." The way people perceive the world is being gradually changed by virtual reality technology, especially for students between the ages of 10 and 17; This technology has a disruptive impact [2].

# **2.2 Introduction of VR Teaching Mode**

Virtual reality teaching is considered an emerging teaching method in the education curriculum, and multiple studies have shown that virtual reality technology has a positive impact on student's academic performance and motivation. Most of the applications of virtual reality technology (VR technology) in primary education have focused on describing and evaluating findings in terms of effectiveness, user experience, usability issues, and student learning performance in the process of primary instructional design [3].

Architecture is a subject that requires people to feel scale. In the process of learning, it is mainly necessary to observe and understand scale, colour, material, and proportion through the eyes. The world of virtual reality can also provide a real experience of scale, get rid of the limitations of the physical world, and present a rich world through a machine.

### **2.3 Feasibility Analysis of VR Education**

1. Upgrade of virtual reality technology

Now virtual reality devices can operate independently, free from the constraints of cables, which also liberates the constraints of environmental sites and makes it possible to teach small classes.

2. Hardware price reduction and popularization

After the acquisition of Oculus, Meta launched the Oculus Qust2 device, which is priced at \$299, enabling ordinary consumers to purchase experience. There is also Pico brand benchmarking in China, and VR devices are becoming increasingly lightweight, enabling more and more ordinary people to have access to this technology.

3. Application of multimedia technology in primary and secondary education

At present, many primary and secondary schools are promoting comprehensive quality education, so with the help of virtual reality technology, students can experience the process of various professions, lay a good foundation for relevant majors, and cultivate interest. And some science experiments with safety risks can bring immersive real experiences through virtual technology.

# **3 Methodology and the Main Procedure**

Our approach focuses on the underlying building of applications and effective integration with VR devices. Consideration is convenient for people to learn and understand quickly.

Build the building application platform and realize the building function. This study combines the intersection of computer science, architecture, and education.

On the application development side, unity3d was selected as the main platform of the whole process operation, and C# language was selected as the programming development language. First of all, I built the module unit in rhino modelling software, exported the module into the unity platform in obj format, and then integrated the model building system module management, DXF file export and other functions on the unity platform.

After the model is built, obtain the generated cutting file and import it to the cutting device. Assemble the cut plates (Fig. 2).

**Fig. 2.** The main steps

#### **3.1 Reality to the Virtual —VR Application Platform Design**

#### **3.1.1 Build the Bottom of the Building System**

Firstly, a 2 m\*2 m cell grid system is established. Grid cells are used to control where blocks are generated. In unity's GridManager, you can enter the desired build width and height. The BlockManager script can adjust the size of the module according to different input parameters, such as a Block 3 m high \*2 m wide \*0.2 m thick.

On the data structure of the GridSystem (Fig. 3), each grid is named GridX and GridZ, respectively, according to the x-axis and z-axis of unity's coordinate system. The grid data structure is divided into three layers, and the GridX of the same row is added to each row according to the row. As shown in the figure below, each GridX record has the column information and the corresponding position of the name, and then multiple rows are added to each layer. The GridX of each layer is then added to a collection called the three-tier list hierarchy. This allows the aspect to read the corresponding position in the program through the sequence.

In the step of placing the block model, the raycast (Fig. 5) method provided by unity3d was used to obtain the grid target hit by the laser ray issued by the VR controller, and the block was created under the object.

Next, the grid is generated based on blocks that can be built in the next layer. After a layer is built, a new layer of GridX and GridZ is generated based on the GridX and GridZ rows and columns of the first layer of blocks. This allows the building to gradually decline from the lower level, which is also in line with the logic of building in a gravity environment. At the same time, the GridCell class is created to record every row and column of GridX and GridZ. This is easy to find the corresponding position in the

**Fig. 3.** Data structure of the grid and the corresponding hierarchy diagram representation

back. During construction, it will detect which grid raycast hits, and after confirming construction, the block will be generated under the grid object. Therefore, it only needs to find the grid with child objects to get the corresponding grid.

Then, rotation and determination of positive and negative rotation (Fig. 2), when each block is placed, there will be a difference between positive and negative, as well as a difference between length and width. Therefore, a record will be made for the direction of the grid and block when it is generated. When a block is generated in the grid, it will judge whether the direction is aligned with the long side so as to ensure that the long side is aligned with the long side. As shown in the figure below, the grid itself will have four directions when it is generated, and the block will also have directions when it is placed. (Fig. 4) matches it in the same direction.

**Fig. 4.** Schematic diagram of the orientation adjustment algorithm of the block enclosure

Finally, in automatic cover determination, after building a floor, the last module returns to the starting position to form a closed ring to generate the floor cover of this floor, based on the range of blocks of this floor, through the GenerateNewFloor method in the GridSystem script. The principle is to detect the maximum and minimum values of blocks in the same column or row to generate the floor within this range. Also, In GridSystem 'CreateStandFloor' method also generates floor non-builds that can stand (Fig. 6).

**Fig. 5.** Camera raycast gets the target grid and generate the block under the corresponding grid

**Fig. 6.** The grid diagram in the multi-layer process is built

#### **3.1.2 Modular Unit Design**

How to reduce cognitive difficulty, improve the richness and increase the fun of architecture from the perspective of children?

In the present architectural design, the modular building has developed to a certain mature stage. In children's toys, building blocks can also be considered modules, so in the early design, there was a choice between voxel unit blocks like Minecraft and modular boards like the SIMS [11].

In the process of module design, the module is also considered.3 m is the common base height. In terms of width, the final consideration is given to both efficiency and free openness. 2 m is selected so that there can be more designs for door and window modules, and rich buildings can be built through different combinations.

When designing the cutting file, based on the ordinary module unit, the jagged card slot is designed on the edge of the module to take into account the assembling problem of children. In this way, the building model can be assembled without glue and other additional tools so as to reduce the assembling difficulty of children (Fig. 7).

**Fig. 7.** Multiple block styles

#### **3.2 Virtual to Reality—Export Cutting Processing Files**

After completing the construction of the virtual model, the digital results are exported from the virtual world to the real world.

First, obtain the location information of the constructed module through unity3d, read a GridCell class mounted on each module, and record the row, col and level of the module. After each layer is built, the information of modules given a direction is recorded, and blocks corresponding to the direction are added to the array. In this way, when generating the DXF file, you only need to read the array of the corresponding direction, find the DXF file corresponding to the module name, use the netdxf library to insert each module separately as an insert block into the generated file, the starting point of the block has been the lower left corner as the origin coordinates. The corresponding three-dimensional model and DXF coordinate system data need to have a transformation formula, and transformation formula algorithm, as below.

$$\mathbf{x} = \mathbf{a} \ast \mathbf{col}, \mathbf{z} = \text{height} \ast \mathbf{Layer}, \mathbf{y} = \mathbf{b} \ast \mathbf{row} \tag{1}$$

where:

**x, y, z**: is the X-axis, Y-axis, and Z-axis coordinate position in the DXF file. **a**: is the constant of the unit length of the grid system. **b**: Is a constant of the unit width of the grid system. **Col**: indicates the number of columns of the block. **Layer**: number of layers of a block **row**: number of rows of a block.

#### **Algorithm1**

Identifying the Outer Contour of Assembled Blocks

1.Load the DXF file for each block.

2.Extract the polyline data from the DXF file for each block.

3.Store the polyline data for each block in a data structure.

4.Loop through each row, column and layer of the blocks.

5.Translate and rotate the polyline data based on the block's position and orientation.

6.Combine the polyline data for each block in the same row, column and layer to form the outer contour.

7.Repeat steps 4 to 6 for all rows, columns and layers to form the complete outer contour for the assembled blocks.

A file is generated for each elevation orientation. The corresponding files are saved in the project directory. After the generation of the cutting file and then opening with the CAD file, according to the location of the block, can separate the different planes of the block. Using the Outside Border Stroke Tool, draw the cut line with another centre line (Fig. 8).

# **4 Conclusion and Discussion**

The popularity of virtual reality technology has brought many possibilities to architectural education. Combined with the immersive feeling in virtual reality, it can increase the fun and authenticity in architecture and provide a more immersive experience environment. It breaks through the traditional teaching mode of staying in books and PPT and

**Fig. 8.** Generated module unit line draft and assembly model

inputs related knowledge of architectural construction into a VR virtual environment, which will greatly improve teaching efficiency.

If TV and mobile phones are a medium to know the world, VR virtual technology is a platform for children to experience, which can provide a more free and imaginative world. This operating system will be separated from the original virtual reality and reality, through the design of the two links, in line with the children's science education process. This building system can customize the design template, give more freedom to children in the design, and ensure that the built results can have enough richness and completion. To promote children's interest in building construction and a general understanding of the construction process.

Future improvement direction: Multiple children will be sought for the experiment. By analyzing the data on children's construction, we can make statistics on children's cognition of architecture and analyze the effect of virtual reality architectural education, as well as their preference and satisfaction with construction from the perspective of children (Fig. 9).

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **The Embodied Interaction with XR Metaverse Space Based on Pneumatic Actuated Structures**

Beibei Zang1, Tianjun Wang2, and Dan Luo3(B)

 Academy of Arts & Design, Tsinghua University, Beijing 10084, China Tisch School of the Arts, New York University, New York 10012, United States School of Architecture, University of Queensland, Brisbane, Australia<EMAIL_ADDRESS>
**Abstract.** This paper is based on our exploration of building an integrated-sensory XR interactive system breaking through the sensory boundary between physical environment and metaverse via pneumatic wearables. In short, our exploration mainly focuses on the following two aspects. Firstly, this research has adapted pneumatic-actuated wearable devices to satisfy the needs of clothing comfort and embodied interaction concurrently by exploring the downsized body-scale pneumatic system and programmable soft materials. Secondly, this research explores the possibility of using digital wearables as the linkage of physical body and XR metaverse to enrich the interaction between XR metaverse and physical environment, aiming at the real-time synchronization of physical wearer's and his virtual avatar in XR system.

**Keywords:** Embodied Interaction · Pneumatic Actuated Structures · XR Metaverse Space

# **1 Introduction**

Digitality has become ubiquitous, permeating every realm of our lives [1]. The boundaries of the human self may extend beyond the physical body, and the consciousness of those extended boundaries has been driven by the development of the outside world [2]. It's important for humans to adapt to the more profound sense of being humantechnological symbionts rather than the merely superficial sense of combining fresh and wires [3]. Since how humans interact with the world is greatly influenced and shaped by the tools used by them [3], wearables are considered as the second skin of people to inter-act directly and broadly with the built environment [4].

Pneumatic-actuated soft robotics has been a widely studied field recently with diverse applications [5]. Via pneumatic actuation, soft robotics has become increasingly accessible [6], and widely researched for application in biomimicry [7]. Furthermore, with the metaverse concept announced by Facebook, virtuality has been meant to have constant and seamless integration with existing physical reality [8]. Therefore, this research uses pneumatic actuated structures as smart garments to improve the comfort of digital wearables as the second skin of people. By using pneumatic actuated structures and physical sensors, this study will try to explore the digital wearables as the linker of physical body and XR metaverse, to enhance the XR sensory and physical comfort concurrently.

Contrasting with other digital bionic structures merely working with formal similarities. This study used pneumatic wearables as the representation of the latent living process embodied in creatures, like heart beating, blood flowing and breathing, to reconstruct the interaction between humans and their surrounding environments. For example, the pneumatic system combined the physical dynamics of plants and the flexible structures of human muscles. At the same time, the wearer's data and the signals from the surrounding environment can be transferred via wearables, triggering the interaction in both the physical and virtual system. In this way. A new dynamic of extended reality has been built with the wearables as an integrated interface of sensing and externalizing. Due to the bulkiness of the air pump of a traditional industrial pneumatic system, in this study, the pump and power system has been further downsized, so that it can meet the size restriction of the body-scale pneumatic system. Last but not least, with the synchronization of the pneumatic actuated wearables communicating via XR metaverse, the comfort of the physical body and the richness and vividness of the senses in the virtual world are simultaneously satisfied, providing prototypes verifying the feasibility for future application (Fig. 1).

**Fig. 1.** The embodied interaction with XR metaverse space on pneumatic actuated structures: metaverse space and pneumatic wearable

#### **2 Methodology and Prototype**

#### **2.1 XR Interaction Design—Physical Interaction of Digital Wearable**

Cognitive philosopher Andy Clark raised the idea in 1998 that human beings are best regarded as an extended system, a coupling of biological organisms and external resources [2]. His ideas coincide with the status of the metaverse today: more than 4.6 billion people can access the virtual worlds of the metaverse via smart phones, laptops, desktops, headsets or consoles [8]. However, communication with the virtual environment is based more on visual content, such as virtual reality platforms providing an engaging and immersive environment [9], or 3D virtual worlds for communication via PC and smart phones [10].

As researchers have concluded, the main feature of metaverse is a twofold link between the virtual and physical worlds: (a) behavior in the physical world influences the experience in the virtual one and, (b) behavior in the virtual world influences the experience in the real one [11]. Touching as a channel for a great variety of information has always been regarded as a crucial site for mediating social perceptions [12] and comprehensive environmental perception [13].

This project explores how the wearables interact with extended reality as an integrated interface of sensing physicality and externalizing XR experience. As a means of telecommunications, the combination of electronic components and sensors for data transferring between the human body and digital system has already been explored in the above-mentioned research. Therefore, instead of adding new applications of telecommunications into the field of human-computer interaction, this project is more about how to introduce the richness and vividness of the XR senses and physical wearable experiences.

In this research, digital wearable devices break through the sensory boundary between physical wearers and its digital avatars. The body movement and heartbeat change of the physical wearer are collected via three-axis acceleration sensor, heartbeat sensor and infrared distance sensor. The movement of the physical wearer is synchronized with its digital avatar, as the touch between digital avatars will cause haptic pressure changes of the physical wearer via its digital pneumatic wearable device in further application.

At the same time, pneumatic wearables visualize and externalize the latent life process of the physical wearer. The digital wearables' inflation and deflation according to the wearer's heartbeat rhythm, trigger changes in the bionic shape of the wearables and thereby change the tactile sensation of the wearer. Wearable devices draw on the techniques of three-dimensional tailoring and the study of muscle composition. Through the tailoring of pneumatics and clothing, the wearer's muscle dynamics during walking can also be creatively represented via wearables in both physical and XR worlds (Fig. 2).

**Fig. 2.** Interaction flow

In the current research of this project, only movement and heartbeat have been introduced into XR interaction and biotic visualization. In future applications, more scenarios of extended reality can be provided by richer collection of physical data such as environmental data of the wearer.

#### **2.2 Pneumatic Actuated Structures and Soft Material**

Under the spread of the digital environment and the popularity of digital devices, the fusion of traditional design and interactive technology has been accelerated in various fields [14], such as the application of 3D print in fashion design pioneered by Iris Van Herpan [15].

However, the above mentioned applications of digital technology devices have downplayed the typical characteristics of traditional fields, such as the basic requirements for comfort in the field of clothing [16]. Therefore, in this research, we have adapted the application of pneumatic systems and programmable materials to meet the basic needs of wearables for comfort and convenience.

#### **2.2.1 Programmable Soft Materials**

The heavy and expensive fabrication of the rigid body [5] hampers its efficiency and flexibility in body-device fit [17]. Therefore, silicone as a thoroughly explored material for soft robotics [18] has been used in this project as a programmable soft material. By computational design supported by Rhino and Grasshopper, we made a variety of 3D molds for silicone casting and fabricated the silicone airbags precisely according to their computed structures. Furthermore, through the combination of programable fabrication and pneumatic system, the deformation of functional garments can be assembled precisely to fit different modes of human activities. Last but not least, TPU has been fabricated as an alternative soft material for creating heat sealable sheets and laminated airtight layer [19]. Via patten prototyping designed in CAD and fabricated with laser cutting, the TPU airbag has also become programmable in the generating process from material to structure (Figs. 3 and 4).

#### **2.2.2 Pneumatic System**

The actuation system consists of three parts: Arduino toolkits for telecommunications between physical and digital space, sensors embedded in wearables, and pneumatic system for controlling the deformation of wearables. This section will introduce the downsized body-scale pneumatic system.

Air pump is controlled by two sets of solenoid valves which have three states (Inflate, Deflate and Hold), to meet the deformation requirement of the wearables via the inflation and deflation of airbags. For safety consideration, the solenoid valve selects the model SMC's S070C-SDG-32 powered by low voltage DC connecting with battery box.

Due to the bulkiness of the air pump of a traditional industrial pneumatic system, this project refits the air port of the portable air pump and connects it to the corresponding set of solenoid valves. Therefore, the entire pneumatic system only needs one portable air pump, two sets miniature electromagnetic valves and an Arduino toolkit connected with sensors. Through the design of digital wearable, the pneumatic system is organized into a pocket-sized box, which satisfies the portability and aesthetics of wearables simultaneously (Figs. 5 and 6).

**Fig. 3.** Programable Soft Materials: TPU and silicone and the process of their computed structures supported by Rhino and CAD. The Fig. 3 shows the process of TPU computed structures which was supported by Tongji University College of Architecture and Urban Planning and developed in DigitalFUTURES Shanghai 2018 Workshop

**Fig. 4.** Programable Soft Materials: TPU and silicone and the process of their computed structures supported by Rhino and CAD. The Fig. 3 shows the process of TPU computed structures which was supported by Tongji University College of Architecture and Urban Planning and developed in DigitalFUTURES Shanghai 2018 Workshop

#### **2.2.3 Morphology and Pneumatic Structures**

The combination of biological structure and morphology design has been widely used in various industrial fields, such as the mechanical properties of natural fiber cells [20] and soft elastic tissues of the human body [21].

To satisfy the comfort of the physical body and the vividness of tactile senses simultaneously, the morphology design draws on the dynamics of flexible structures and soft tissues, specifically, muscles as active elastic elements, skin as passive elastic elements and flytrap as pneumatic actuated structures.

The main structure of wearables follows the dynamic of muscle movement, while the holistic structure design takes both the static parts and kinematic joints into consideration.

**Fig. 5.** Pneumatic system design. The pictures on the left show how the pneumatic system is designed. The two pictures on the right show the testing of pneumatic system

**Fig. 6.** An example of pocket-sized pneumatic system with one portable air pump, one pair of two-sets miniature electromagnetic valves that can control two sets of airbags, and an Arduino toolkit which can connect sensors easily

Therefore, the pneumatic structure can be compatible with the daily activities of wearer, even assist the muscles' movement.

The airbag position mapping is generated by Grasshopper from the heat map of body movement. In the original prototype, the pneumatic structures consisted of airbag units fabricated with TPU and teflon after laser cutting. In further iterations, the pneumatic structures are added inflatable muscular structures made of silicone casting. In the composed pneumatic structures, the active parts fitted body movement are composed of muscular silicone structures, while the static parts are added on more biological features, to improve the comfort and richness of wearable devices (Figs. 7 and 8).

#### **2.3 Interaction with XR Metaverse Space**

The real-time synchronization between the physical wearer's behavior and one's virtual avatar is realized by Kinect motion capture via processing, physical sensors and their connections with metaverse space. As introduced in the physical interaction chapter, the wearer's activity data triggers deformation of the wearable device in physical space,

**Fig. 7.** Various prototypes of pneumatic air-bags' structures inspired by muscles and flytrap

and the mirrorly changes of the digital avatar's position and appearance happen concurrently via the linkage of physical data and metaverse space. Furthermore, beyond mere synchronization of the physical wearer's activities, a richer interactive experience has been introduced into metaverse space via HoudiniFX space rendering. Walking and turning of the physical wearer will trigger smoke and light effects in metaverse space. Therefore, the wearer of Digital Wearable can experience extended-sensory interaction between XR metaverse and physical environment with integrated perception of tactile changes and visual effects (Figs. 9, 10, 11 and 12).

# **3 Conclusion and Discussion**

The vision of this research is to build an integrated-sensory XR interactive system breaking through the sensory boundary between physical environment and metaverse. The current prototype has achieved the three goals mentioned above:

**Fig. 8.** Various prototypes of pneumatic air-bags' structures inspired by muscles and flytrap

**Fig. 9.** XR design: the connections between the wearer, the digital wearable and the digital avatar

**Fig. 10.** Real-time synchronization between the physical and digital ones by Kinetic motion capture via Processing

**Fig. 11.** Various experiences of metaverse space made by HoudiniFX

**Fig. 12.** An example of The Embodied Interaction with XR Metaverse Space based on Pneumatic Actuated Structures

Future development will focus on improving the delicacy of sensory transmission and the richness of interactive activities between physicality and extended reality. Specifically:

**Acknowledgements.** This research was supported by Tongji University College of Architecture and Urban Planning and developed in DigitalFUTURES Shanghai 2018 Workshop with team members Shaoting Zeng, Heidi and Junwon Yoon. The authors would like to express great thanks to the instructors and teaching assistants of the workshop, specifically, Behnaz Farahi, Jifei Ou, and Yuqiong Lin. The authors would also like to extend grateful appreciation to MIT Media Lab Tangible Media Group, whose work laid the hardware foundation and motivated the authors to do this research. And special thanks to Mitsuru Isshiki for his help with building 3D characters and metaverse space.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Construction of Recreation Behavior Simulation Model of Public Space in Urban Waterfront—Taking Huangpu River in Shanghai as an Example**

Chunxia Yang and Ming Zhan(B)

Tongji University, 1239 Siping Road, Shanghai, China<EMAIL_ADDRESS>
**Abstract.** This study constructs a multi-agent behavior simulation model to explore the quantitative simulation method of waterfront public space. Taking 6 waterfront public space samples along the Huangpu River in Shanghai as research objects, this study first collects environmental data and pedestrian behavior data through field survey, and then analyzes and processes the data to obtain the Spatial Attraction Weight (SWA) that expresses the relationship between pedestrian behavior and spatial elements. Then, based on the Anylogic platform, the pedestrian agent particles expressing people's characteristics are placed into the simulation environment based on the social force model. They interact in real time to dynamically simulate the pedestrian's behavior. Finally, fitting verification of the preliminary model is carried out. The qualitative comparison and quantitative correlation analysis are combined to enhance the accuracy. The behavior simulation model of waterfront public space built in the study can more realistically represent the pedestrian's behavior. It can realize the scientific prediction of the future use of waterfront space and provide more detailed reference for problem diagnosis and optimization.

**Keywords:** waterfront public space · Recreation behavior · Micro-behavior simulation · Multi-agent

# **1 Research Background**

As a unique space resource, urban waterfront has become an important recreational place in the city because of its continuous, open and water-friendly space characteristics. However, some extensive development and construction have made the urban waterfront public space low in vitality. The current problems are mainly manifested in improper connection of the base plane, single shoreline forms, limited public buildings, and dislocation of supporting facilities. The traditional small-sample survey, empirical design and pre-judgment make the allocation of spatial elements in waterfront areas not fully fit the user behavior preferences caused by waterfront characteristics. It is necessary to find more scientific, quantitative and intelligent analysis and simulation methods to guide the construction of urban waterfront space.

Multi-agent behavior simulation technology is a cutting-edge means of microbehavior simulation. It considers the interaction between individuals and the environment, and is suitable for simulating individual behavior. It can realize full-time dynamic simulation, multi-plan comparison and virtual situation preview. Its full-time interactive dynamic simulation can better present the self-organized behavior of the pedestrian under the influence of complex environmental space elements in the urban waterfront, and the output simulation results can be used as the basis for evaluating the future use of public space.

# **2 Research Object and Research Path**

Based on the Anylogic platform, this study constructs a behavior simulation model to simulate pedestrians' behavior in urban waterfron public spacet. It intuitively and dynamically presents the actual use of waterfront public space, and provides reference for the design, decision-making and management of waterfront public space. According to the principles of convenient accessibility, rich space, and open all day, this study selects six waterfront public space samples which are adjacent to each other in 3 pairs of the Huangpu River in Shanghai as objects for simulation. Six typical time periods (7: 00–9: 00, 9: 00–11: 00, 11: 00–13: 00, 13: 00–15: 00,15: 00–17: 00 and 17: 00–19: 00) are selected for the survey (Fig. 1).

**Fig. 1.** Six sample sites along the Huangpu River

The construction process of the behavior simulation model based on Anylogic platform includes four parts: basic data collection, data analysis and processing, simulation model construction and operation fitting adjustment (Fig. 2).

**Fig. 2.** Research framework

# **3 Research Key Links**

Behavior simulation technology was initially widely used in the simulation of evacuation behavior in indoor spaces such as stations, airports, gymnasiums, etc. In recent years, it has been extended to the simulation of outdoor pedestrian spaces and commercial streets. Currently, most of the relevant studies are rarely applied to the simulation of outdoor public space and daily recreational behavior. In order to carry out targeted simulation research, the following three key links should be defined for the model construction of urban waterfront public space.

#### **How to obtain and quantify the impact mechanism of the interaction between waterfront public space and pedestrian behaviors?**

Firstly, types and characteristics of spacial elements should be analyzed through investigation. Then the classification of waterfront recreational behaviors and their basic characteristics, including pedestrian movement, vision, environmental response and other characteristics, are studied. On this basis, we should explore the impact of waterfront spacial elements on pedestrian behavior and consider how to convert them into impact weight that can be recognized by the model.

This study focuses on the pedestrian behaviors and distribution in a specific waterfront space environment, and uses the spatial element attractiveness weight (SWA), which is an important model construction parameter to quantify the relationship between spacial elements and pedestrian behaviors.

#### **How to combine the advantages of agent model and social force model to build a behavior simulation model?**

Commonly used micro-behavior simulation models include cellular automata model, social force model, agent model magnetic force model, etc. Most of the current research is based on existing models to simulate behaviors with obvious regularities in various spaces. In order to improve the precision of the simulation, some scholars have improved the existing models. For example, Chiung-Hui (2011) improved the agent model based on the theory of visual attention, and used the agent program to represent the shopping behavior rules. Based on the nonlinear exit allocation strategy, Song et al. (2018) improved the social force model to avoid problems such as hitting a wall and distortion of exit capacity in simulation. However, current research rarely combines the advantages of each micro-behavior simulation model to construct a combined model and conduct targeted research. Meanwhile, research on the behavior simulation of complex urban public space lacks a mature combination model construction method.

In this study, the Anylogic platform that supports multi-method combination modeling is selected to build a combination model based on the advantages of the agent model and the social force model. The pedestrian path decision is dominated by the agent particles, which are placed in the simulation environment constructed based on the social force model, and the pedestrian path selection is expressed through the pedestrian behavior chain.

#### **How to conduct fitting analysis and verify the validity of the model?**

Fitting analysis is the key to verify whether the behavior simulation model can truly reflect the actual condition. The fitting method often uses an intuitive comparison between the measured and simulated results, and whether the fitting is effective depends on the empirical observation of the researcher. In recent years, quantitative fitting has gradually received attention, but more attention has been paid to the overall space, and there is a lack of fitting methods for local space. For example, Guo et al. (2014) applied Depthmap and SPSS correlation analysis software to fit the pedestrian behaviors in shopping centers under different spatial organization models. The model predicted the distribution of people flows in the overall space well, but the prediction accuracy of the distribution of local people flow in each layer did not meet expectations.

This study combines qualitative graphical comparison with quantitative correlation analysis, and takes the measured and simulated the number of people attracted by each attraction as the fitting basis to carry out sub-item simulation and fitting by time period and behavior type to improve the accuracy of the model.

# **4 Behavior Simulation Model Construction Process**

# **4.1 Basic Data Collection**

Basic data collection can be divided into environmental element data collection and pedestrian behavior data collection. This study classifies the diverse spatial elements of the waterfront public space into four categories of base plane, shoreline form, buildings, facilities and their subcategories. Using the method of field survey to collect the information of spatial elements. The record of the location and quantity of micro-spatial elements is especially paid attention. Pedestrian behaviors are divided into five categories of viewing, leisure, sports, entertainment, consumption and their subcategories. Pedestrians' waterfront recreation behaviors are diverse and random, with individual differences and obvious time differences. By means of mapping, counting and questionnaire, the information of pedestrian characteristics, behavior types, behavior locations, and behavior duration are obtained.

# **4.2 Data Analysis and Processing**

Compared with commercial streets or other urban public spaces, the urban waterfront public space has more complex and diverse spatial elements. Pedestrian behavior is more diverse and random due to the comprehensive influence of various environmental spacial elements. The distribution of pedestrian in waterfront shows that different spacial elements have distinct attraction to various pedestrian behaviors, that is, the probability of pedestrian choosing different spacial elements to carry out behaviors is different. The attraction weight of each space element expresses the relationship between spatial elements and pedestrian behaviors, which is an interactive parameter that can be identified by the model platform. This study adopts the Spatial Attraction Weight (SWA) in the waterfront public space to quantitatively express the regular relationship between the spatial elements and pedestrian behaviors in the waterfront public space, to establish the relationship between waterfront environment, pedestrian behaviors and the simulation model parameters. The calculation formula is as follows:

$$\text{SWA}\_{\text{(xy)}} = \frac{N\_{\text{(SA)}}}{N\_{\text{(T)}}}$$

N(SA) in x–y period, the total number of people of certain recreation behaviors in certain spatial elements.

N(T) in x–y period, the total number of people of all recreation behaviors in the waterfront public space.

#### **4.3 Simulation Model Construction**

Based on the Anylogic platform, the multi-agent behavior simulation model is constructed for six samples. Its modeling process can be divided into four parts: model operation process design, simulation environment modeling, pedestrian agent particles modeling and pedestrian behavior process modeling. Next, taking DongchangWaterfront as an example, the model construction process is introduced in detail.

#### **4.3.1 Model Running Process Design**

The process of pedestrian recreation behavior in waterfront public space can be understood as a kind of space-time change, with the dual attributes of time and space. The waterfront recreation process of pedestrians has no relatively unified model, and their behaviors in the waterfront public space are full of randomness and diversity. Recreational behavior is not only a movement in space. In the process of being attracted to stay in some areas, pedestrians are actually engaged in recreational behavior although they do not move in space, which needs to be expressed through time changes.

The model operation process is divided into five parts: particle emission, direction determination, particle susceptibility determination, particle attraction determination and recreation end determination.

#### **4.3.2 Simulation Environment Modeling**

Firstly, the CAD base map is imported into the Anylogic platform, and various spatial elements of base level, shoreline form, buildings, facilities and their subcategories in the site are translated based on the spatial markup module (Space Markupin) of the Anylogic to build a simulation environment. As static agents, the spatial module has three basic attributes: service radius, pedestrian capacity and attraction weight, which are defined through the Parameter module of the Anylogic agent library (Fig. 3).

**Fig. 3.** Simulation environment of Dongchang waterfront

# **4.3.3 Pedestrian Agent Particle Modeling**

In order to fully consider the characteristic differences of people, each pedestrian is simulated as a unique agent particle with its own attributes. Besides the three initial attributes of gender, age and behavior type, the pedestrian agent has four basic attributes: vision range, planned recreation time, basic speed and element perception radius. Pedestrian particles with different gender and age are given different basic attributes. In the Anylogic, the Parameter of agent library is used to express the four fixed basic attributes. Variable of agent library is used to express the dynamic attributes of pedestrian agent, such as moving speed, moving direction and moving time, which may change in the running of the model.

### **4.3.4 Pedestrian Behavior Process Modeling**

Pedestrians' behavior process corresponds to the model operation process. Based on the behavior module of Anylogic, a pedestrian behavior chain that presents the behavior characteristics of random selection of pedestrians along the waterfront is constructed to express the path selection and decision-making process of pedestrians in the waterfront (Fig. 4).

**Fig. 4.** Behavior chain

The pedestrian particles are put into the simulation environment built by the spatial markup module that based on the social force model. At each step of the continuous process, pedestrian agent particles can comprehensively simulate various attractions and repulsion forces in the space and make autonomous behavior decisions. It dynamically presents the behavior process of pedestrian waterfront recreation and the spatial distribution of pedestrians.

# **4.4 Fitting Analysis and Validity Verification**

On the basis of completing the preliminary construction of the combination model, the sub-item scenario simulation is carried out, and the results of the field survey are fitted by time period and behavior type.

# **4.4.1 Time-Period Fitting Analysis**

In this study, the preliminary model simulation results are compared with the survey and measurement results, and Spatial Attraction Weight (SWA) of each element is adjusted through multiple simulations to make the simulation results highly consistent with the actual situation. The fitting method is the same for each time period, and the final result basically conforms to the actual situation is obtained through fitting (Fig. 5).

**Fig. 5.** Summary of simulation results of Dongchang waterfront in different periods

On the basis of qualitative fitting, quantitative fitting analysis is added to improve the simulation accuracy of local space. SPSS Statistics is used to analyze the bivariate correlation between the simulated data and the measured data of the attraction points in each period.

There are a total of 62 spatial attraction points in four categories in Dongchang Waterfront. The actual number of people attracted in the site survey and the simulated number of people attracted by the 62 attraction points in each time period are counted. The bivariate correlation analysis of the measured data and the simulated data of each attraction point in each time period is carried out through SPSS Statistics. The Pearson coefficient is the minimum of 0.840 (7:00–9:00) in the correlation analysis results. The maximum is 0.989 (17:00–19:00), which is greater than 0.6. And the significance level is 0.000, far less than 0.01, which proves that the model is effective (Table 1).

# **4.4.2 Behavior Type Fitting Analysis**

On the basis of fitting analysis in different periods, the fitting of different types of behaviors is added. The model built in this study can simulate particles of a certain

**Table 1.** Correlation analysis of the number of people attracted by attraction points

type of behavior separately. We choose the 15:00–17:00 period with the largest number of people to compare the simulation results from the field survey results of each type of behavior. Taking the viewing behaviors as an example, the simulation results show that the viewing behaviors mainly take place at the wooden path, the rest platform and the cruise ship wharf on the concave shoreline, which is basically consistent with the measured results (Fig. 6).

**Fig. 6.** Partial enlarged figure of viewing behavior particle simulation result

Similar to the time-period fitting, the actual and the simulated number of people attracted for different types of behaviors at 62 attraction points in Dongchang Waterfront during this period are counted and analyzed. The results show that the Pearson coefficient was 0.703 for viewing behaviors and 0.986 for leisure behaviors, both of which are greater than 0.6. The significance level is 0.000, which proves that the model is effective.

#### **4.4.3 Model Validation**

The adjacent Dongchang Waterfront, which is basically the same as the surrounding environment of Dongchang Waterfront, is selected to verify the effectiveness of the model. The Spatial Attraction Weight (SWA) of the spatial elements after fitting and adjustment is given to the spatial elements in the adjacent waterfront to verify model validation. The simulation results are highly correlated with the measured results, which prove that the model is effective. The Spatial Attraction Weight (SWA) obtained by simulation can provide reference for the simulation of other similar waterfront.

# **5 Model Application**

Applying the above modeling methods, this study builds a behavioral simulation model for the waterfront section of the North Bund of Shanghai and carries out sub-scenario simulation. Firstly, the public space problem is diagnosed through simulation and optimization and improvement measures are proposed. Then this study improves the spatial elements such as various base planes, shorelines, buildings and facilities, and carries out the combination rehearsal and comparison, and puts forward the best recommendations with different orientations. Finally, an approximate combination is selected and compared with the reconstructed condition of the waterfront section of the North Bund which verifies the feasibility of the behavior simulation modeling analysis method (Fig. 7).

**Fig. 7.** Actually measured after transformation and pedestrian distribution of optimized and rehearsed before transformation

# **6 Conclusion**

This research innovatively applies behavior simulation technology to the study of recreational behavior in urban waterfront public space, which extends the application scope of behavior simulation. The behavior simulation model constructed in this study can perform multi-scenario rehearsal for the allocation scheme of spacial elements in the urban waterfront, and realize scientific prediction of future use. It provides a more detailed design basis for diagnosis and optimization of waterfront. The scene preview of the optimization scheme can avoid inefficient repeated renewal to a certain extent.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Estimating the Impacts of Seasonal Variations of Streetscape on Dockless Bike Sharing Trip with Street View Images and Computer Vision**

Yongqin Zhao1, Sijia Wang1, Difan Chen2, Kaijie Huang3, Siyuan Zhang4, Waishan Qiu5, and Wenjing Li6(B)

 Independent, Shanghai, China Independent, Missouri City, CA, USA The New School, New York City, USA Huazhong University of Science and Technology, Wuhan, China Cornell University, New York City, USA The University of Tokyo, Tokyo, Japan<EMAIL_ADDRESS>
**Abstract.** A significant portion of the cycling experience is influenced by the streetscape, and this impact varies throughout the year. The temporal dynamic of streetscape has been neglected in most previous studies, including urban public mobility route choices. This paper examines the correlation between dockless bike sharing and streetscape as well as spatial elements in different seasons using a large amount of GPS bike trajectory data collected by LIME. The study shows that: (1) DBS volume is significantly influenced by seasonal streetscape factors such as roads, cars, sidewalks, tree, and vegetation color; (2) How significantly these seasonal factors affect DBS volume differs in summer and autumn; (3) In both summer and autumn models, non-seasonal factors like mixed land use score, street network connectivity, etc., are significant. Some non-seasonal factors only impact the DBS volume in one season; (4) When adding subjective perception to models of both seasons, model explanatory does get improved very slightly.

**Keywords:** Seasonal variation · Dockless bike sharing · Street view image · Computer vision · Built environment

# **1 Introduction**

Bikeshare promotes sustainable travel, health benefits, and economic growth (Qiu and Chang 2021). Dockless bikeshare (DBS), compared to the docked bikeshare system, is getting more popular in the last decade due to benefits like accessibility and convenience (Gu et al. 2019).

There are observed research gaps in DBS seasonal study: (1) the development of mobile applications and cashless mobile payment have make bike sharing usage even more prevalent (Guo et al. 2022). However, as a new mode of transportation, DBS has received less attention than docked bike sharing (Guo et al. 2022). (2) A majority of precedent studies of DBS focus mostly on where a trip starts and ends, rather than the cycling experience itself. (3) limited examination of how seasonal streetscape affects cycling. Although a small number of studies consider temporal scale, yearly comparisons (Li 2021) offer limited help in comparing seasons, and studies addressing the association between seasonal climate and bike sharing didn't examine other seasonal environmental characteristics(Li and Kamargianni 2017).

The study (1) provides a quantitative study of DBS focusing on perceived environmental elements along the journey, (2) integrates SVI and CV to estimate how seasonality of street built environment impacts DBS usage at a fine spatial scale. (3) considers previously ignored seasonal environmental features like vegetation color.

# **2 Data and Methodology**

### **2.1 Study Area and Methodology**

Our study area includes the Town of Ithaca and a few nearby neighborhoods (Fig. 1).

**Fig. 1.** Study area

Figure 2 illustrates the framework of this study: (1) Using GPS bike trajectory data from LIME, a DBS system in Ithaca USA, we computed Seasonal Weighted Rides (SWR) to capture the cycling volumes of road segments in summer and autumn. (2) We collected SVIs in summer and autumn with Google SVI API. (3) We used PSPNet to compute the view ratio indices of streetscape elements, and used Mask R-CNN to count the number of streetscape objects. We also computed CV indicators (color deviation, L, A, B values in CIELAB color space) to present the seasonal color change of these three variables: tree, plant, and grass. (4) We quantified four subjective perception scores (accessibility, ecology, enclosure, scale) of street environment in summer and autumn with ML. (5) We collected and computed non-seasonal variables (typical POI, landmark POI, infrastructure, road type, land use mixed score, street network connectivity, terrain). (6) With OLS regression models, the seasonal environment attributes are comprehensively analyzed with their impacts on DBS volumes in summer and autumn.

**Fig. 2.** Research framework

#### **2.2 Data and Processing**

#### **2.2.1 Dependent Variable: DBS Data and Seasonal Weighted Rides (SWR)**

LIME provided us with the DBS trips data for this study, accessible through an API. LIME's app was used to collect the data. After the users agreed to LIME's terms and conditions for using the services, LIME recorded and analyzed their journeys. The dataset does not include identifiable individual information. To clean the data, trips that started or ended outside the Greater Ithaca area (Qiu and Chang 2021), trips with distances shorter than 0.05 miles or 264 feet (Qiu and Chang 2021), and trips with durations less than 3 min or more than 120 min were removed. The validated dataset has 102,178 trip records.

To make cyclists' choice and preference of routes comparable across different seasons, we have to capture the popularity of road segments in each season with Seasonal Weighted Rides (SWR). This conversion can be found in Eq. (1).

$$SWR\_j = SR\_j / (\sum\_{j=1}^{n} r\_j) \tag{1}$$

For segment *j*, *SRj* represents the volume of rides on it in this particular season, *n* represents the number of segments with rides on them in this particular season, and *n j*=1 *rj* represents the total number of rides on all segments in Ithaca during this particular season. The *SWRj* that is aggregated to the sampling points*i* is the dependent variable of this study, and has the precision of street segments, so sampling points on the same segment would have the same value for one season.

To reduce the bias caused by data sparsity, we removed segments with a total annual volume of less than 500 riders (approximately 1.5 riders a day) from the aggregated data. Then we sampled points every 25 m in segments with lengths more than 50 m (which is too short). Therefore, 671 out of 10508 road segments were selected, and 2,082 sampling points were obtained from them.

#### **2.2.2 Independent Variable: Street View Measuring**

Google Street View (GSV) is only available in summer (Jun, July, Aug) and fall (Sept, Oct, Nov) in our study area, and out of 2,082 sampling points, there are only 1,170 points having GSV in both seasons. With GSV panorama ID and Street View Download 360 software (Street View Download 360, n.d.) we download the panorama in both summer and fall of each point. To extract the count of elements in a panorama SVI, we use a Mask-RCNN pre-trained on COCO dataset with ResNet-50 backbone. Then we use Pyramid Scene Parsing Network (PspNet) with pre-trained model psp\_resnet101\_ade to conduct image segmentation.

Undesired segmentation distortions might occur near the top and button of a panorama image, so we unwarp the panorama into images in 6 directions (Forward, Back, Left, Right, Up, and Down) with py360convert package and extracted the four directions at eye-level: forward(F), back(B), left(L), and right(R) (Fig. 3).

From percentage that the pixels of the specific visual element take-up of the total pixels of an SVI we calculated the visual ratio of an element in the image. Not all objective view indices will be input to the regression model after the Variance Inflation Factor (VIF) test, only 20 out of 28 visual ratio are kept, including: tree, road, grass, car, streetlight, wall, building, sidewalk, earth, water, plant, awning, van, person, bridge, railing, bicycle, minibike, ceiling, chair.

To study color and change of street greenness from urban cyclists' perspective, we used the CIELAB colorspace and extracted PSPNet pixels for three types of greenness. Converted from RGB to CIELAB using the Python-colormath library, we calculated average A and B values for each pixel and the standard deviation from actual values. To eliminate potential interference, we eliminated the L value (brightness) as SVIs are taken in different conditions.

We evaluate street perceptions using a 300-point pre-labeled dataset and an ML framework developed by related research (Qiu et al. 2023; Su et al. 2023) to evaluate the subjective perceptions score of the streetscape: Accessibility (accessibility to

**Fig. 3.** Panorama object detection with Mask\_RCNN, unwarping(in 4 directions: F, R, B, L), and semantic segmentation with PSPNet. Left: Summer SVI. Right: Autumn SVI

activities, attractions, and amenities), Ecology (detecting living organisms, animals, plants, humans, and their physical environment), Enclosure (the degree to which buildings, walls, trees, and other vertical elements define streets and public spaces visually), and Scale (human-sized and proportional elements). The dataset was collected from a crowdsourced visual survey of an expert panel and includes SVI input variables and 4 perceptual scores as output labels. 75% (225) of the dataset is used for training and 25% (75) for testing. Multiple ML algorithms are used, including K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Random Forest (RF), Gaussian Process (GP), Gradient Boosting Regression (GB), ADA boost, and Bagging Regression. GP is chosen as the optimal model for predicting the target perceptions (Table 1).

#### **2.2.3 Independent Variable: Non-seasonal Variables**

We use Open Street Map (OSM) to get typical POI, infrastructure (transit facility), road types, and landmarks POI. The buffer zone radius is 500m for typical POIs, 100 m for infrastructure, and 1000 m (typical 5 min bike rides) and 3000 m (typical 15 min bike rides) for landmark POIs.

We use land use data collected from Tompkins County Open Data Portal (Land Use and Land Cover 2015; Tompkins County Open Data Portal, n.d.), and use a method

**Table 1.** Performance of GaussianProcessRegressor (GP) predictions

originally developed to calculate the evenness of distribution of the area of different use types (Frank et al. 2005). The calculation is shown in Eq. (2).

$$\text{LandueMix} = (-1) \ast \left(\sum\_{l=1}^{n} p\_l \ln(p\_l)\right) / \ln(n) \tag{2}$$

With *pi*, the proportion of land use *i* within the 500m buffer is divided by the total area within the 500 m buffer, while *n* represents the number of exclusive land use types within the 500 m buffer. This formula would calculate the mixed land use level within range 0–1, a higher value means a higher level of mixed land use.

We use Depthmap X to run the space syntax calculation. However, we finally choose Connectivity and Angular Integration (with segment length weighted, or SLW) after removing space syntax score with high multicollinearity (VIF > 10). Angular Integration (SLW) has two metric radius parameters: 250 and 1000 m. For a particular segment, Connectivity describes quantity of other segments it connects with, and Angular Integration (SLW) captures accessibility and how close(integration level) of this segment is to all others in terms of the sum of angular change(Angular Integration Space Syntax—Online Training Platform, n.d.).

We use high resolution(1 m) Digital Elevation Model (DEM) from United States Geological Survey (USGS) collected in May 2020. The medium value of slope within 5 m of sampling point along the road is chosen as the slope value.

#### **2.3 Model Architecture**

We start with a simple OLS model Eq. (3).

$$Y\_l = \alpha + \sum\_m X\_{(l,m)} \beta\_m + \epsilon\_l \tag{3}$$

*Yi* is the dependent variables of the sampling point. *X* is the independent variables that explains *SWR*. β is the coefficient of variable *m* that reveals how and to what extent variable *m* is related to *SWR*. Constant term α refers to the average *SWR* when all other variables are zero. Error term *i* captures elements that influence the *Y* but are not included in *X* .

A baseline model (M1\_Summer, M1\_Autumn) with significant variables was constructed using all variables except subjective perception. VIF was calculated in the whole process and only variables with VIF less than 10 are kept. Then all 4 perception scores are added (M2\_Summer and M2\_Autumn) (Table 2).

**Table 2.** Dependent variable, independent variable, and method of different models

# **3 Result and Discussion**

#### **3.1 Models Performance and Diagnosis Results**