The NLU module aims to identify the intent of a user's query. Most existing applications in NLU were developed to detect exact keywords to understand user's intentions. This study utilized semantic and syntactic analysis of natural language processing. Compared to the detection of exact keywords, the NLU allows for more flexible word choices for natural language queries. Some research reported that Deep Neural Networks (DNN) methods can improve the flexibility of natural language understanding (Kepuska and Bohouta, 2018; Packowski and Lakhana, 2017). However, training a deep learning language model for an SDS requires a large amount of dialogue data, and such data is limited in the construction industry. Therefore, this study implemented natural language processing methods for the NLU module. The NLU module was developed to identify content keywords and classify them. This study focused on directly extracting attribute information from "*IfcBuildingElement*", so the speech query is targeting *IfcDoor*, *IfcWindow*, *IfcWall*, etc. For example, consider a non-BIM expert project manager who is reading a PDF drawing, and the manager would like to know the height information for a window with a known tag. The manager used the spoken natural language query "What is the height of the window 356213?" which contains the content keywords "height", "window", and "356213". These content keywords would be identified and classified into *attribute word* (i.e., "height"), *type word* (i.e., "window"), and *name phrase* (i.e., "window 356213") by this module. The developed algorithm uses natural language processing methods, like sentence tokenization and Part-of-Speech tagging, to analyze the semantic contents and syntactic structure of the textual natural language query (see Figure 2). After tokenization and Part-of-Speech tagging, content keywords (i.e., noun, adjective, and cardinal number) are identified. Although the *attribute word* (i.e., "height") and *type word* (i.e., "window") are all nouns, the *type word* is within a prepositional phrase (PP) (i.e., of the window). The *name phrase* (i.e., "window 356213") consists of *type word* with a cardinal number or adjective. The identified keywords are used to locate the target IFC data and generate the corresponding textual natural language response.

Figure 2: NLU Algorithm for iBISDS Framework

### **3.3 Building Information Extraction Module**

The BIE module extracts the target building information, for example, "the height of window 356213". The building information model in the iBISDS framework is an IFC-STEP file. Although IFC is an open-source specification, the data structure of an IFC-STEP file is complex. To parse an IFC-STEP file and improve the efficiency of information extraction, this study developed an open-source Python package – *IfcReader*, which was published in GitHub (https://github.com/wangningstar/IfcReader). The BIE module was developed based on *IfcReader*, which can extract organized IFC data based on the IFC schema. The detailed algorithm for the BIM module is shown in Figure 3. The identified *type word* (i.e., "window") by the NLU is used to find the target IFC entity type (i.e., *IfcWindow*). To locate the target type, all IFC entity types of *IfcBuildingElement* within an IFC-STEP file are extracted into a list. The next step is to check whether the *type word* or its synonym is within the list. This study implemented WordNet to find synonyms of the *type word*. After the target IFC entity type is located, the next step is to locate the target IFC entity. The BIE module checks whether the *name phrase* (i.e., "window 356213") is a substring of one IFC entity. If so, the next step is to extract the queried data from the target IFC entity by checking whether the *attribute word* (i.e., "height") or its synonym is a substring of an attribute name of the target IFC entity. *IfcReader*  is utilized to extract all attribute names of the target IFC entity into a list based on the IFC4 ADD2 TC1 schema. For example, the list of attribute names of an *IfcWindow* is "['GlobalId', 'OwnerHistory', 'Name', 'Description', 'ObjectType', 'ObjectPlacement', 'Representation', 'Tag', 'OverallHeight', 'OverallWidth', 'PredefinedType', 'PartitioningType', 'UserDefinedPartitioningType']". The BIE module uses the *attribute word* (i.e., "height") or its synonym to match the target attribute 'OverallHeight' of the target *IfcWindow* and extract the corresponding value. The target value is utilized to generate a natural language response.

Figure 3: BIE Algorithm for iBISDS Framework

#### **3.4 Natural Language Generation Module**

The NLG module aims to generate a natural language sentence based on the structured information from the NLU and BIE modules. The algorithm of this module was developed in previous efforts (Wang and Issa, 2020b). The NLG module used Part-of-Speech to generate a natural language sentence. The generated natural language sentence is a template-based pattern: "The" (determiner) <*attribute word*> "of" (preposition) "the" (determiner) <name phrase> "is" (verb) <extracted IFC data> <unit>. The pattern is following the basic structure of English syntax: noun phrase and verb phrase. "The" (determiner) <*attribute word*> "of" (preposition) "the" (determiner) <name phrase> is a noun phrase, while "is" (verb) <extracted IFC data> <unit> is a verb phrase. Classified keywords from the NLU module and extracted IFC data from the BIE module were utilized as content words to generate the sentence. The <unit> information is extracted from the entity *IfcConversionBasedUnit*. The <unit> can be imperial or metric which is predefined in the IFC file by users. If the <extracted IFC data> is a cardinal number that is not "1", the <unit> will be changed to the plural format. For example, the generated natural language for "the height of window 356213" is "The height of the window 356213 is 4 feet". In addition, the <unit> is unnecessary, if the <extracted IFC data> is not a cardinal number. For example, if a user queried the type information of a window, the generated natural language would be "The type of the window 356255 is fixed:36" x 48".".

## **3.5 Text-to-Speech Module**

The TTS module aims to convert the generated textual natural language into a speech response back to a user. The TTS module is the key to achieve the "spoken" part of an SDS. Many companies and organizations have been developing TTS technologies, e.g., Google, IBM, and Microsoft. With the adoption of deep learning methods, the sound quality and naturalness of synthesized speech of existing TTS technologies have been improved (Joo et al., 2020; Sun et al., 2020). Therefore, the iBISDS framework adopted the Google TTS engine to convert a textual response into a speech one. After the conversion, the generated speech was played by the developed TTS module. The synthesized speech response is a more convenient way for construction practitioners to obtain the queried information response.

#### **4. Verification and Discussion**

A Python-based prototype program was developed based on the iBISDS framework. The prototype was used to verify the logic and algorithm of each module in the framework. The integrated development environment (IDE) was the PyCharm community 2020.3 version, and the Python interpreter was Python 3.8.5 distributed by Anaconda. This study developed a preliminary Graphical User Interface (GUI) based on *Tkinter* for the client-side of the iBISDS (see Figure 4). The prototype program is a server-based system. The server-side generates a port, and the client-side can request a service through the port. The server-side was developed based on the iBISDS framework: ASR, NLU, BIE, NLG, and TTS. The ASR module implemented the Google speech recognition service from the Python package *speech\_recognition* to convert a speech query into a textual one. A desktop microphone was used to receive the speech query. The NLU module utilized the *nltk* Python package to Part-of-Speech tag the textural natural language query. This module was developed based on syntactic analysis to identify and classify keywords. The developed package *IfcReader* was used to parse an IFC-STEP file and extract organized data in the BIE module. Also, the synonyms function was developed based on the WordNet with the *nltk* version to find all synonyms of *attribute word* and *type word*. The BIE was developed based on a substring matching algorithm to extract the target building information. The NLG module utilized the natural language pattern to generate a textual response. The <unit> information was also extracted by *IfcReader* if the extracted building information is a cardinal number. The TTS module adopted the Google Textto-Speech package *gTTS* to convert the generated textual response into a speech stored in a .mp3 file and played it for users.

The preliminary results indicated that the iBISDS framework yielded valid results. The ASR module correctly converted a speech query into a textual one. The NLU correctly identified and classified keywords within the textual query. The target IFC data could be extracted by the BIE module. The NLG and TTS modules generated a natural language response and converted it into speech back to users. Compared to the detection of exact keywords, the iBISDS framework enabled a flexible option for speech queries. Construction project team members can use speech to query iBISDS, and the iBISDS will provide a speech response back to them. Also, a smartphone can be the client-side of iBISDS. Users can receive responses to queried building information to support construction activities on the construction site. The iBISDS framework still has some limitations. The NLU module implemented Part-of-Speech and syntactic analysis methods to identify and classify keywords within the textual query, because of the lack of dialogue training data for the construction industry. The NLU uses tags to locate target building elements which restricts the syntactic structure of the speech query. To provide a more flexible NLU, training data should be collected and labeled for deep learning use in the future. This study focused on directly extracting attribute information from *IfcBuildingElement* without computation and reasoning. After the development of a deep learning-based NLU, future research should explore quantity information extraction and ontology-based reasoning. For example, the BIE module with ontology-based reasoning can locate the window in room 101 instead of using tags to locate target building elements. The developed iBISDS is modularized which means each module in iBISDS can be substituted for by future efforts.

Figure 4: Preliminary Graphical User Interface of iBISDS

#### **5. Summary and Conclusions**

Existing building information acquisition methods require BIM users to spend more time studying the BIM database structure and software manipulation. Compared to conventional SQL or SPARQL-based IE methods, the speech-based IE is expected to be more acceptable to BIM users. With the development of conversational AI technologies, SDS has become increasingly more popular in human daily life. Therefore, this research developed an iBISDS framework with a focus on directly extracting attribute information from *IfcBuildingElement*. The basic architecture of the iBISDS consists of five modules: Automatic Speech Recognition, Natural Language Understanding, Building Information Extraction, Natural Language Generation, and Text-to-Speech. Detailed algorithms for each module were developed in this study. A Python-based prototype program was developed to verify the iBISDS framework and algorithms. The preliminary results indicated that the iBISDS framework is valid for recognizing natural language speech queries, extracting the target building information, and generating the corresponding speech responses. The iBISDS enables a machine to use speech natural language to respond to a user's speech query. The framework is just the start of iBISDS, and it currently has some limitations. To provide a more intelligent SDS for BIM, deep learning methods and ontology will be implemented in future research. Also, the iBISDS framework provides a basic architecture for developing an SDS for other research areas, such as construction safety. For example, an SDS for Occupational Safety and Health Administration (OSHA) standards can provide spoken safety instructions for on-site construction laborers via speech queries. It is expected that the iBISDS framework will lead to further adoption of conversational AI technologies in the AECO industry.

#### **References**

Eastman, C., Fisher, D., Lafue, G., Lividini, J., Stoker, D., Yessios, C., 1974. An Outline of the Building Description System. Research Report No. 50.

Isyanto, H., Arifin, A.S., Suryanegara, M., 2020. Design and Implementation of IoT-Based Smart Home Voice Commands for disabled people using Google Assistant, in: Proceeding - ICoSTA 2020: 2020 International Conference on Smart Technology and Applications: Empowering Industrial IoT by Implementing Green Technology for Sustainable Development. Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ICoSTA48221.2020.1570613925

Joo, Y.S., Bae, H., Kim, Y.I., Cho, H.Y., Kang, H.G., 2020. Effective Emotion Transplantation in an End-to-End Text-to-Speech System. IEEE Access 8, 161713–161719. https://doi.org/10.1109/ACCESS.2020.3021758

Jucks, R., Linnemann, G.A., Brummernhenrich, B., 2018. Student Evaluations of a (Rude) Spoken Dialogue System Insights from an Experimental Study. Adv. Human-Computer Interact.

Kajinami, K., Nishimura, R., Kitaoka, N., 2018. Construction of dialog database for development of spoken dialog breakdown detection methods, in: ICAICTA 2018 - 5th International Conference on Advanced Informatics: Concepts Theory and Applications. Institute of Electrical and Electronics Engineers Inc., pp.91–95. https://doi.org/10.1109/ICAICTA.2018.8541273

Karan, E.P., Irizarry, J., Haymaker, J., 2016. BIM and GIS Integration and Interoperability Based on Semantic Web Technology. J. Comput. Civ. Eng. 30, 04015043-1-04015043–11. https://doi.org/10.1061/(asce)cp.1943-5487.0000519

Kepuska, V., Bohouta, G., 2018. Next-generation of virtual personal assistants (Microsoft Cortana, Apple Siri, Amazon Alexa and Google Home), in: 2018 IEEE 8th Annual Computing and Communication Workshop and Conference, CCWC 2018. Institute of Electrical and Electronics Engineers Inc., pp.99–103. https://doi.org/10.1109/CCWC.2018.8301638

Kobayashi, Y., Yoshida, T., Iwata, K., Fujimura, H., Akamine, M., 2019. Out-of-Domain Slot Value Detection for Spoken Dialogue Systems with Context Information, in: 2018 IEEE Spoken Language Technology Workshop, SLT 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., pp.854–861. https://doi.org/10.1109/SLT.2018.8639671

Lin, J., Hu, Z., Zhang, J., Yu, F., 2016. A Natural-Language-Based Approach to Intelligent Data Retrieval and Representation for Cloud BIM. Comput. Civ. Infrastruct. Eng. 31, 18–33. https://doi.org/10.1111/mice.12151

Liu, H., Lu, M., Al-Hussein, M., 2016. Ontology-based semantic approach for construction-oriented

quantity take-off from BIM models in the light-frame building industry. Adv. Eng. Informatics 30, 190–207. https://doi.org/10.1016/j.aei.2016.03.001

Packowski, S., Lakhana, A., 2017. Using IBM Watson Cloud Services to Build Natural Language Processing Solutions to Leverage Chat Tools, in: Proceedings of the 27th Annual International Conference on Computer Science and Software Engineering, CASCON '17. IBM Corp., USA, pp.211–218.

Palconit, M.G.B., Formentera, A.L., Aying, R.J., Dianon, K.J.A., Tadle, J.B., Dadios, E.P., 2019. Speech Activation for Internet of Things Security System in Public Utility Vehicles and Taxicabs, in: 2019 IEEE 11th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management, HNICEM 2019. Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/HNICEM48295.2019.9073370

Park, Y., Kang, S., 2019. Natural Language Generation Using Dependency Tree Decoding for Spoken Dialog Systems. IEEE Access 7, 7250–7258. https://doi.org/10.1109/ACCESS.2018.2889556

Ralston, K., Chen, Y., Isah, H., Zulkernine, F., 2019. A voice interactive multilingual student support system using IBM watson, in: Proceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019. Institute of Electrical and Electronics Engineers Inc., pp.1924–1929. https://doi.org/10.1109/ICMLA.2019.00309

Sacks, R., Eastman, C.M., Teicholz, P.M., Lee, G., 2018. BIM handbook : a guide to building information modeling for owners, designers, engineers, contractors, and facility managers, Third edit. ed. John Wiley & Sons, Inc., Hoboken, New Jersey.

Sun, G., Zhang, Y., Weiss, R.J., Cao, Y., Zen, H., Rosenberg, A., Ramabhadran, B., Wu, Y., 2020. Generating Diverse and Natural Text-to-Speech Samples Using a Quantized Fine-Grained VAE and Autoregressive Prosody Prior, in: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. Institute of Electrical and Electronics Engineers Inc., pp.6699–6703. https://doi.org/10.1109/ICASSP40776.2020.9053436

Wang, H.-H., Boukamp, F., Elghamrawy, T., 2011. Ontology-Based Approach to Context Representation and Reasoning for Managing Context-Sensitive Construction Information. J. Comput. Civ. Eng. 25, 331–346. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000094

Wang, N., Issa, R.R.A., 2020a. Ontology-Based Integration of BIM and GIS for Indoor Routing, in: Construction Research Congress 2020. Tempe, AZ.

Wang, N., Issa, R.R.A., 2020b. Natural Language Generation from Building Information Models for Intelligent NLP-based Information Extraction, in: Ungureanu, L.C., Hartmann, T. (Eds.), EG-ICE 2020 Workshop on Intelligent Computing in Engineering. Universitätsverlag der TU Berlin, Berlin, pp.275–284. https://doi.org/10.14279/depositonce-9977

Wen, T.H., Young, S., 2020. Recurrent neural network language generation for spoken dialogue systems. Comput. Speech Lang. 63, 101017. https://doi.org/10.1016/j.csl.2019.06.008

Yamamoto, K., Tamagawa, A., Nakagawa, S., 2019. Evaluation of real robot agent interface for spoken dialogue system, in: 2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019. Institute of Electrical and Electronics Engineers Inc., pp.694–695. https://doi.org/10.1109/GCCE46687.2019.9015424

Zhang, L., Issa, R.R.A., 2013. Ontology-Based Partial Building Information Model Extraction. J. Comput. Civ. Eng. 27, 576–584. https://doi.org/10.1061/(asce)cp.1943-5487.0000277

## **A Design Recommender System: A Rule-based Approach to Exploit Natural Language Imprecision using Belief and Fuzzy Theories**

Lucian-Constantin Ungureanu Technische Universität Berlin, Germany<EMAIL_ADDRESS>
**Abstract.** This paper proposes a framework to handle the natural language imprecision using belief and fuzzy theories. The goal is to provide a stepping-stone towards the creation of more flexible natural language interfaces between humans and computers to support design. The focus is on the natural language, as a result of verbal communication in design collaboration. Language imprecision in such setups is essential to design creativity, and the current design support systems do not accept such imprecise input. The proposed approach drafts a set of rules which are then implemented as a computer program to showcase the achieved behaviour. The achieved behaviour needs to be validated with designers, but it showcases how procedural power of the ambiguous and vague language can be harnessed to generate a population of design alternatives as recommendations.

### **1. Introduction**

We can still argue that nowadays computers cannot compute with words. It is still the imprecise character of the natural language that poses significant computational challenges. Language imprecision is defined as a lack of clarity and precision (Zhang, 1998). On the one hand, it allows listeners to hypothesize about meaning, creating various interpretations. On the other hand, it allows speakers to cope with the lack of information and knowledge for a given situation. In design and engineering, language imprecision is seen as an integral part which enables creativity. Previous studies (Ungureanu & Hartmann, 2021) show that designers' language, when communicating design changes, is imprecise. In design studies, researchers highlighted the benefits of language imprecisions, such as helping creativity (Durrant et al., 2018; Wiegers et al., 2011) and allowing for interpretative flexibility (Glock, 2009). However, at the intersection between humans and computers, human language imprecision hinders the human-computer dynamic. Computers are seen as inflexible, passive and formal (Dossick & Neff, 2011), characteristics which require precise formalization of the input. To break this barrier, flexible human-computer interfaces must allow users to use imprecise language to interact with computers, especially in specialized environments such as design sessions where language imprecision is a key characteristic to a creative process.

Imprecise language expressions encapsulate sufficient information to provide recommendations (Jurafsky & Martin, 2013). To this end, imprecise language such as vague expressions, convey procedural meaning (Jucker et al., 2003). Recent interest on more natural interfaces between computers and humans resurrected the interest of creating systems that make sense of language vagueness when searching for products (Papenmeier et al., 2020), or which provide guidelines on how to interpret approximate numerical expressions (Lefort et al., 2017). Other than acknowledging its presence in design and engineering there is very little done in the direction of allowing the design support systems to harness the power of imprecise and vague language. One remarkable effort (Abualdenien & Borrmann, 2020) in this direction proposes a method to visualize vagueness in design models. While this focuses on information representation, to my knowledge, there is no previous research looking into how to incorporate ambiguous and vague input into a design model.

This paper proposes an approach using belief theory and fuzzy logic to support the creation of a design recommendation system using as input ambiguous and vague language expressions. The research presented in this paper focuses on a very small subset of possible problems related to language ambiguity and vagueness with the purpose of providing a precedent in the direction of handling imprecise input. This paper adopts a listener perspective, which is characterized by the following information flow: Statement Interpretation Action Modified Situation (Sowa, 1999). The research in this paper aims to covers aspects related to interpretation and action and their mathematical formalization. It proposes the creation of a set of rules that serve as possible routes of dealing with the ambiguity related to the attribute naming when communicating a change and vagueness related to the attribute value with the focus on continuous numerical attributes. Moreover, the paper adopts a design science research method with problem-centered initiation. This allowed to create a prototype implementation of the proposed approach, to evaluate it, to identify possible limitations, and propose future research directions based. The behavior achieved after the implementation of the proposed framework is showcased on a small theoretical parametric model of a room. The rest of the paper is structured as follows: next section present some more insight into language imprecision in design and computational approaches to handle it. Next section introduces the proposed approach, followed by the research methodology, implementation, and results. The final two sections of the paper discuss the implementation and the results and present the conclusions of the study.

#### **2. Language imprecision in design and computational approaches**

Language imprecision in design can take various forms. It is mainly related on how designers use words to covey their ideas. Some of the previous studies focused on designers' language and their use of metaphors and hedges (Christensen & Schunn, 2007, 2009), polysemy (Georgiev & Taura, 2014), slang and jargon (D'Souza & Dastmalchi, 2017; Kleinsmann & Valkenburg, 2008). In short, language imprecision can take various forms classified as fuzziness, uncertainty, vagueness, possibility, and probability (Raskin & Taylor, 2014). In spoken language, these correspond to various intentional commitments a speaker might make. In the communication between designers, especially when communicating design changes, it has been shown that ambiguity and vagueness are present (Ungureanu & Hartmann, 2021). To this end, a human-computer interface needs to be flexible enough to handle such cases of language ambiguity and vagueness.

Some of the approaches in the field of linguistics propose various strategies to deal with language ambiguity and vagueness. While the humans are wired to make sense of the imprecise input, information systems need precise information to work. When it comes to ambiguity, a common strategy is to identify the meaning of the word in context through disambiguation. Some of the state-of-the-art approaches in this direction are proposed by Pasini & Navigli (2020) and Wang et al. (2020). This solved half of the problem. Once the real meaning of the word is identified how to recommend something to the user? Considering the use of the word size, a disambiguation method might identify to which of the artefact attributes it is connected, such as in the case of a room to length, width, and height. Yet, it might not provide directions regarding which one of these parameters to change to reach the design change impact envisioned by the speaker. Language vagueness can also have multiple facets. In connection with design changes communication, vagueness is linked to communicating the extend of a change (Ungureanu & Hartmann, 2021). Examples of possible linguistic expressions are "a little bit bigger", "a little bit wider", "about" (Khan & Tunçer, 2019; Ungureanu & Hartmann, 2021; Wiegers et al., 2011). To handle language vagueness, Zadeh, (1996) proposed the use of fuzzy logic, terming this as computing with words. While this approach is far from being an automated one, it relies on the knowledge linked to the vague expressions used in the day-today conversations and allows computing using this knowledge formalized in the form of rules. It is one of the common approaches used in control systems. For instance, a rule like "*if(cold) then (Increase temperature)"* is a simple example of a rule in a control system using fuzzy logic. The closest to come to deal with language imprecision in design is (Lawson & Loke, 1997) who proposed the use of sliders to allow designers to manually vary attributes' values. In this paper, we proposed the use of belief theory to handle language ambiguity and the use of fuzzy logic to handle language vagueness. The approach we proposed in this paper is presented in the next section.

#### **3. Proposed Approach**

In this paper, the perspective we embrace over the design is following the reasoning laid down by the systems-in-systems theory. Figure 1 showcases a very simple example of a rectangular room. A very first step within this approach is to establish the boundaries of the system. In this case, even if the room is usually part of a higher-level system such as a building, we see the room as the main system. This system can be decomposed further in smaller elements such as the ones shown at Level 2 (e.g., wall, floor, ceiling). At each level, no matter the decomposition granularity, a system has specific attributes driving its design. Another boundary related aspect we consider in this paper, is that our focus is on the numerical attributes associated with a system at a given

level. We can represent this mathematically as () = { , = 1 ≤ ≤ } where D(S) is the design function of the system S, dependent on the set of numerical attributes {Ai}. Considering the example of our room, this formulation is exemplified as *D(Room) = {A1: length, A2: width, A3: height}*. In the same way, the design function of the wall can be formulated as *D(wall) = {A1: length, A2: height, A3: thickness}*. The design is developed as a succession of states = 1, , where j is the current state and j+1 a future possible state. In each state j an attribute A<sup>i</sup> has a value vij. These values have a predefined range called the universe of discourse. The universe of discourse of a given attribute value can be mathematically represented as a bounded interval such as ∈ [ , ]. We can mathematically capture this logic as following:

$$D\_{\!\!\!\!\!}(\mathbb{S}) = \{ A\_{\!\!\!\!\/} = \{ \upsilon\_{\!\!\!\!\/} \in \left[ \upsilon\_{\!\!\!\!\/}^{\min}, \upsilon\_{\!\!\!\!\/}^{\max} \right] \}, 1 \le i \le n \}.$$

The transition of the design from current state Dj(S) to a next state Dj+1(S) is subject of a transformation operator and a goal G to be achieved (Eastman, 1969). If the goal G lacks a precise formulation the transformation is termed by Eastman (1969) as a ill-defined problem. Eastman (1969) did not considered the lack of precise formulation of the transformation operator. In this paper, we consider both aspects. The goal is to formulate an information processing pipeline to handle the cases when the definition of the goal and transformation operator lack precision. As mentioned in the introduction, we focus on the verbal communication of the designers, when communicating changes related to various parts of the design. Designers' communications include the natural language formulation of a specific transformation and the desired goal. In most of the cases, their communication lacks precision – ambiguous and vague linguistic expressions are used instead of precise formulations (Ungureanu & Hartmann, 2021). In this paper, we focus on the ambiguous communication of the ATTRIBUTE subjected to changes and vague communication of VALUE change of the attribute. A precise natural language formulation of a design change implies that a given ATTRIBUTE is named and the extent of the change (i.e., VALUE) are clearly communicated. An example, related to our example in Figure 1 will be "*increase the length of the room with 300mm.*" (E1) In information processing, a such precise formulation can be captured in the form of an IF-THEN rule.

*Rule (1): Given the system S: IF (ATTRIBUTE in {* , = 1 ≤ ≤ *} AND VALUE is PRECISE) THEN f(ATTRIBUTE, VALUE)*

where, f() is a change function dependent on the communicated operation. Given the example, the function f() is an increase function indicated by the action word "*increase*" and the design state *j* function:

$$\mathsf{ID\_{\mathcal{I}}(\mathsf{Rosom})} = \{\mathsf{A\_1} \colon \mathsf{Length} = \mathsf{v}\_{1\circ \mathsf{I}} \colon \mathsf{A\_2} \colon \mathsf{width} = \mathsf{v}\_{2\circ \mathsf{I}} \colon \mathsf{A\_3} \colon \mathsf{height} = \mathsf{v}\_{3\circ \mathsf{I}}\}$$

is transformed to the next *j+1* state as following.

$$\mathsf{D}\_{\mathsf{j}+1}\{\mathsf{R}\mathsf{o}\mathsf{om}\} = \{\mathsf{A}\_{1}\colon \mathsf{l}\mathsf{Length} \text{ is } \mathsf{A}\mathsf{T}\mathsf{R}\mathsf{T}\mathsf{B}\mathsf{U}\mathsf{E} = \{\mathsf{v}\_{1\circ 1\circ 1} = \mathsf{v}\_{1\circ 1} + \mathsf{V}\mathsf{A}\mathsf{L}\mathsf{U}\mathsf{E}\},$$

$$\mathsf{A}\_{2}\colon \mathsf{w}\mathsf{l}\mathsf{d}\mathsf{t}\mathsf{h} = \{\mathsf{v}\_{2\circ 1\circ 1} = \mathsf{v}\_{2\circ 1}\}, \ \mathsf{A}\_{3}\colon \mathsf{h}\mathsf{s}\mathsf{l}\mathsf{g}\mathsf{h}\mathsf{t} = \{\mathsf{v}\_{3\circ 1\circ 1} = \mathsf{v}\_{3\circ 1}\}$$

Ambiguity and vagueness are two natural language characteristics associated with language imprecision. The expression "*increase the size of the room*" (E2) is a frequent expression used by designers (Ungureanu & Hartmann, 2021). In this case, the ambiguity gives the listener the freedom to hypothesize about a combination of attributes which can be changed to increase the generic attribute *size*. We update Rule (1) to cover the case when the attribute is ambiguous using the belief theory.

*Rule (2): Given the system S: IF (ATTRIBUTE not in {* , = 1 ≤ ≤ *} AND VALUE is PRECISE) THEN* ( )*{f(Ai, VALUE), }, where* ∑ ≤ 1 =1

βi called degree of belief, and ( ) denote the k-combinations of the n attributes with k taking values from 1 to n. The sum of all the assigned degrees of believe need to be smaller or equal to 1 (Yang et al., 2006). For the room example, the rule (2) can be detailed as following:

*Rule (2): Given the system S: IF (size not in {* , = 1 ≤ ≤ *} AND VALUE is PRECISE)THEN* ( ) { 1: ℎ = {1 + 1 = 1 + }, <sup>1</sup> = 0.45 2: ℎ = {2 + 1 = 2 + }, <sup>2</sup> = 0.45 3: ℎℎ = {3 + 1 = 3 + }, <sup>3</sup> = 0.1 }

Considering the attribute space of the room example, the solution space provided by the rule (2) is ( 3 1 ) + (3 2 ) + (3 3 ) = 7 consisting of solutions taking individually or combinations of each parameter in the attribute space. The assignment *β1 = 0.45* means that there is *45%* belief that the attribute *A1:length* is the attribute to be increased. Considering the low degree of belief assigned to β3 = 0.1 this can be argued from the perspective of the domain knowledge. Usually, the height of a room is not changed individually; in most of the cases the designer will change the height of the entire floor, and consequently the height of all the rooms on the floor.

We can take further the expression (E2) and, besides ambiguity, we can add the expression "*a little bit*" as a vague indicator of the extent of the change. We get the expression "*increase a little bit the size of the room*" (E3). As shown by the previous research (Khan & Tunçer, 2019; Ungureanu & Hartmann, 2021; Wiegers et al., 2011) designers use qualitative quantifiers to communicate the extend of a change. To handle this case of linguistic imprecision we propose the use of fuzzy logic. Thus, we consider each attribute Ai as a linguistic variable, rather than a numerical variable. In fuzzy logic, a linguistic variable is defined as:

### = 〈, (), , , 〉

where: x is the variable name, T(x) is a set of terms, U is the universe of discourse, G a set of syntactic rules, and M a set of semantic rules. Figure 2 exemplify all the elements of a linguistic variable and provides an example of fuzzification. A linguistic variable such as *Length* can be communicated through linguistic syntax using a set of soft linguistic terms. This set of terms, through semantic rules are represented as fuzzy subsets distributed over the universe of discourse of the linguistic variable using some membership functions. Using the fuzzy subsets, we can represent the crisp number 3 as a fuzzy number [0.66, 0.34, 0.00].

Figure 2: Linguistic variable; example fuzzification length = 3

Table 1: Knowledge base matrix for rules creation

In our case, the universe of discourse for an attribute, usually defined by designers, takes values between [min, max]. Different membership functions can be defined to indicate, based on domain and situational knowledge, the membership of various values to each term from the set. The simplest function which can be defined is the triangular one (Figure 2). We adopt this approach for the fuzzification of the attribute's values. Each attribute will have a universe of discourse between minimum and maximum values assigned by designers with a certain degree of membership to a linguistic term. Table 1 presents a proposed knowledge base to build the set of rules to be used for inference. For example, a rule which can be created based on Table 1 is: IF (Current value is Small) AND (input is "*a little bit bigger*") THEN output is ALBB. The membership functions are assumed as trapezoidal functions following the percentages distributions indicated in Figure 3. Moreover, Figure 3 shows how the inference is done based on this rule. The output is represented by the grey area under the "*a little bigger*" membership function. The final step is to define a strategy for the defuzzification. A common approach is to consider the centre of gravity of the grey area, approach adopted in this paper.

Figure 3: Example inference

To this end, rules (1) and (2) can be updated to reflect the cases when the VALUE is imprecisely communicated by the user.

**Rule (3):** Given the system S: **IF** (ATTRIBUTE in {Ai, 1≤i≤n} **AND** VALUE is not PRECISE) **THEN** f(ATTRIBUTE, g(VALUE))

**Rule (4):** Given the system S: **IF** (ATTRIBUTE not in {Ai, 1≤i≤n} **AND** VALUE is not PRECISE) **THEN** ( ){f(Ai, g(VALUE)), βi}, where ∑ ≤ 1 =1

where g(VALUE) represents the fuzzy function.

### **4. Research Methodology**

This paper employs a design science research method following the framework proposed by Peffers et al., (2007). Figure 4 presents the problem-centered research method which consists of five main components. The main objective of the current paper is to develop a prototype recommender system that exploits the imprecise input provided by users and provides as output various design alternatives. Figure 5 presents the architecture adopted for the envisioned voice interface. It takes the form of a voice-based command system making use of the state-of-theart pipeline for automatic speech recognition and natural language processing provided by Amazon Alexa. The output from Alexa represents the content of a set of slots. These slots, together with the intent are predefined for this implementation via Alexa Developer console. The slots represent the semantic elements related to {ATTRIBUTE, VALUE}, where ATTRIBUTE can be any of the attributes of the parametric model presented in Figure 6. As parametric modeling tool, the proposed implementation is using Dynamo BIM, a design automation and parametric modeling tool. The implementation of the proposed approach is made as a view extension for Dynamo, programmed using C# and deployed as a \*.dll class library. DynAlexa starts a Http server and based on user query to an Alexa device it receives an Alexa post request input as an intent with its slots. The intent serves to distinguish the conditional cases, and the content of the slots are sent further for processing using the implementation of the proposed approach. The four generic rules of the proposed approach were coded in this implementation using FLS C# library.

Figure 4: Design Science Research Method with Problem-Centered Initiation

Figure 5: Architecture of the envisioned system

### **4. Implementation and results**

The implementation of the proposed framework led to a first working prototype which allow the user to use imprecise language to input changes in a parametric model. The end-to-end verification highlighted that the created prototype behaves as expected, although the implementation itself helped identify several areas which need further attention from the research community. The implementation made use of a simple parametric model of a room shown in Figure 6. The parametric model has three attributes defined as controlling (input)parameters, namely width, length, and height. The universe of discourse of each parameter is present in Figure 6. The degrees of belief (DoBs) for each input parameter were generically defined in advanced for the cases when the user communicate an ambiguous attribute such as size as shown in the same figure.

For the cases when an ambiguous attribute is named by the user in an utterance such as "*increase the size of the room*", the user received a set of combinations sorted in descending order based on the general degree of belief (gDoB) calculated for each possible combination as the average of the degrees of belief (DoBs) of the parameters part of the combination. The user has the possibility to click on any of the provided combinations and to visualise in real time how the model is changed.

Figure 6: The simple parametric model of a room used for the implementation

Figure 7: Generated list of design options based on ambiguous attribute naming

Table 2: Results of Fuzzy Logic implementation when the user provides vague qualifiers. The values after each attribute are the [min, max] values. The values after each value assigned to a vague qualifier represent the increase (+) and the decrease (-) relative to the initial value

The results of the fuzzy logic implementation are shown in Table 2. As shown in Figure 3, the fuzzification of the attribute value is made using triangular membership functions, and the defuzzification of the vague qualifiers is made using trapezoidal membership function. Figure 8 shows an example of how the values in the Table 2 were calculated. The trapezoidal functions are defined as ration of the intervals [min, current value] and [current value, max], depending on the operation. If the current value is far away from the min/max values, rate of increase/decrease will be quite significant. If the current value is close to the min/max values, then the rate of increase/decrease will be small. This behaviour is visible in Table 2, and it is because the membership functions are defined using percentages of the remaining intervals from min/max to the current value. This can be seen in the values withing the brackets. Increasing a little bit the length from 9.5m produces an increase of +2.4m. Increasing a little bit the height from 2.7m produces an increase of only +0.1m.

Figure 8: Example showing the steps followed to "*increase*" "*a little bit*" the "*length*" from 9.5m

#### **5. Discussions and Conclusions**

Current paper proposes an approach based on belief theory and fuzzy logic to handle the language imprecise input from the user when communicating design changes. The proposed approach can be implemented on any voice-based or text-based user interfaces with the purpose of harnessing imprecise language to provide design recommendations. The implementation showcases a promising direction which allows the user to explore various scenarios when ambiguously communicating the name of an attribute. Moreover, the system allows the users to use vague qualifiers to indicate the extend of a change. The degrees of belief help sort the various scenarios. The fuzzy logic setup allows to convert the vague qualifier into numerical values. The implementation also highlights certain areas for future research.

First and foremost, the entire process needs to be supported by a knowledge base. A parametric model might have various attributes available associated with various parts of the design, named length, width, height. A knowledge base will allow for inferences, so that the system do not confuses for example the length of the bedroom with the length of the living room. Regarding the degrees of belief, future research needs to establish strategies regarding their definition. The approach adopted in this paper to manually define them is tedious and time consuming. When a user refers to the size of a room, to what extend they refer to length, width, or height? Maybe in this direction one might have to think of developing a rule-based system based on domain knowledge. Another option can be a system that learns based on users' preferences. Regarding the fuzzy logic approach, one might need to define the level of granularity of a such system when it comes to the fuzzification and defuzzification in equal measure. In this paper we used a set of three linguistic sets for the input variable, and a set of another three linguistic sets for the defuzzification of the operations. Is this enough or does it require a higher level of granularity? Moreover, when it comes to the defuzzification, is it enough to return a single value based on a method such as centre of gravity or is it better to output a population of values? In addition, the current implementation contains only a small number of rules for a limited number of linguistic variables. Increasing the number of linguistic variables will considerably increase the number of rules needed to be implemented. Smart approaches are needed to identify and handle in a unique manner similar linguistic variable such as *larger* versus *bigger*. The same with ambiguous references to attributes. Our assumption is that computing similarities between various words might serve as an automated approach to assign degrees of belief and to link a generic attribute name to one linked to an artefact. Future research needs to also be done to expand the framework for other types of attributes such as non-numerical ones (e.g. categorical, shape attributes).

While little has been done in this direction, the current paper serves as a steppingstone in the direction of allowing the users to imprecisely provide input to computer. The framework allows for the creation of a design support system that allows designers to get real-time feedback on their design moves. In this direction, research also need to be done to identify whether this approach is beneficial for the design process. To close, the proposed system has the potential to reduce computers inflexibility and allow designers to actively incorporate them in the design process.

#### **References**

Abualdenien, J., & Borrmann, A. (2020). Vagueness visualization in building models across different design stages. Advanced Engineering Informatics, 45(April), 101107. https://doi.org/10.1016/j.aei.2020.101107

Christensen, B. T., & Schunn, C. D. (2007). The relationship of analogical distance to analogical function and preinventive structure: The case of engineering design. Memory and Cognition, 35(1), 29–38. https://doi.org/10.3758/BF03195939

Christensen, B. T., & Schunn, C. D. (2009). 'Putting blinkers on a blind man' Providing cognitive support for creative processes with environmental cues. Christensen, B. T. & Schunn, C. D. Tools for Innovation, 48–74.

D'Souza, N., & Dastmalchi, M. (2017). "Comfy" cars for the "awesomely humble": Exploring slang and jargons in a cross-cultural design process. Analysing Design Thinking: Studies of Cross-Cultural Co-Creation, 1993, 311–330. https://doi.org/10.1201/9781315208169

Dossick, C. S., & Neff, G. (2011). Messy talk and clean technology: communication, problem-solving and collaboration using Building Information Modelling. The Engineering Project Organization Journal, 1(2), 83–93.

Durrant, A. C., Kirk, D. S., Moncur, W., Orzech, K. M., Taylor, R., & Pisanty, D. T. (2018). Rich pictures for stakeholder dialogue: A polyphonic picture book. Design Studies, 56, 122–148.

Eastman, C. (1969). Cognitive processes and ill-defined problems: A case study from design. International Joint Conference on Artificial Intelligence: IJCAI, January 1969, 669–690.

Georgiev, G., & Taura, T. (2014). Polysemy in Design Review Conversations. Design Thinking Research Symposium, 2003, 1–19. http://docs.lib.purdue.edu/dtrs/2014/Identity/2

Glock, F. (2009). Aspects of language use in design conversation. CoDesign, 5(1), 5–19.

Jucker, A. H., Smith, S. W., & Lüdge, T. (2003). Interactive aspects of vagueness in conversation. Journal of Pragmatics, 35(12), 1737–1769. https://doi.org/10.1016/S0378-2166(02)00188-1

Jurafsky, D., & Martin, J. H. (2013). Speech and language processing: pearson new international edition. Pearson.

Khan, S., & Tunçer, B. (2019). Speech analysis for conceptual CAD modeling using multi-modal interfaces: An investigation into Architects' and Engineers' speech preferences. AI EDAM, 1–14.

Kleinsmann, M., & Valkenburg, R. (2008). Barriers and enablers for creating shared understanding in co-design projects. Design Studies, 29(4), 369–386.

Lawson, B., & Loke, S. M. (1997). Computers, words and pictures. Design Studies, 18(2), 171–183. Lefort, S., Zibetti, E., Lesot, M. J., Detyniecki, M., & Tijus, C. (2017). Dimensions for automatic interpretation of approximate numerical expressions: An empirical study. International Conference on Intelligent User Interfaces, Proceedings IUI, 107–117. https://doi.org/10.1145/3025171.3025174

Papenmeier, A., Sliwa, A., Kern, D., Hienert, D., Aker, A., & Fuhr, N. (2020). 'A modern up-to-date laptop' - vagueness in natural language Queries for Product Search. ArXiv, 2077–2089.

Pasini, T., & Navigli, R. (2020). Train-O-Matic: Supervised Word Sense Disambiguation with no (manual) effort. Artificial Intelligence, 279, 103215. https://doi.org/10.1016/j.artint.2019.103215

Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45–77. https://doi.org/10.2753/MIS0742-1222240302

Raskin, V., & Taylor, J. M. (2014). Fuzziness, Uncertainty, Vagueness, Possibility, and Probability in Natural Language. 2014 IEEE Conference on Norbert Wiener in the 21st Century (21CW), 1–6.

Sowa, J. (1999). Knowledge representation: logical, philosophical and computational foundationsNo Title. Brooks/Cole Publishing Co.

Ungureanu, L., & Hartmann, T. (2021). Analysing frequent natural language expressions from design conversations. Design Studies, 72, 100987. https://doi.org/10.1016/j.destud.2020.100987

Wang, Y., Wang, M., & Fujita, H. (2020). Word Sense Disambiguation: A comprehensive knowledge exploitation framework. Knowledge-Based Systems, 190, 105030. https://doi.org/10.1016/j.knosys.2019.105030

Wiegers, T., Langeveld, L., & Vergeest, J. (2011). Shape language: How people describe shapes and shape operations. Design Studies, 32(4), 333–347.

Yang, J. B., Liu, J., Wang, J., Sii, H. S., & Wang, H. W. (2006). Belief rule-base inference methodology using the evidential reasoning approach - RIMER. IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans, 36(2), 266–285. https://doi.org/10.1109/TSMCA.2005.851270 Zadeh, L. A. (1996). Fuzzy logic = computing with words. IEEE Transactions on Fuzzy Systems, 4(2), 103–111. https://doi.org/10.1109/91.493904

Zhang, Q. (1998). Fuzziness-vagueness-generality-ambiguity. *Journal of Pragmatics*, *29*(1), 13–31.

## **Towards the Adoption of Vision Intelligence for Construction Safety: Grounded Theory Methodology based Safety Regulations Analysis**

Numan Khan, Muhmmad Khan, Seungwon Cho and Chansik Park\* Chung Ang University, Seoul, South Korea<EMAIL_ADDRESS>
**Abstract.** The construction safety rules play a vital role in mitigating accidents and fatalities at the construction site. Many researchers are currently devoted to monitoring the rule compliance using vision intelligence-based approaches; however, such systems are still not yet mature to be applied in construction job sites. A single autonomous source's job site safety control is a non-trivial task that needs a detailed analysis of safety rules for a compact vision intelligence-based system development. This paper proposes Grounded Theory Methodology (GTM) to systematically classify the safety rules for implementation using vision intelligence technology. The rules are classified into four groups based on the open coding, axial coding, and selective coding approach: (1) Before Work, (2) With Intervals, (3) After Work (4) During Work. The proposed GTM based model linked further with the scene capturing sources such as single scene capturing through smartphones for the rules required: (a) before work and after work, (b) periodic scene capturing using robots, and (c) drones for the rules needed with intervals and (d) CCTVs for the safety rules to continuous monitoring of safety.

#### **1. Introduction**

The construction industry is infamous due to the adverse situation of accidents and fatalities happening in the construction job sites. These enormous accidents have made the construction industry one of the most unsafe sectors (Nath, Behzadan and Paal, 2020). The Bureau of Labor Statistics (BLS) in 2017–18 reported 5,250 fatal work injuries, recorded a relative increase of 2 percent concerning the data reported in 2017, (BLS, 2019). The non-fatal injuries also remained at the peak; for instance, the non-fatal accidents recorded during 2017 were 79810, accounting for 9% of the total non-fatal accidents in construction (BLS, 2017). These accidents are expensive and can be prevented by taking excellent safety measures (Park, Lee and Khan, 2020). Numerous safety policy procedures and comprehensive safety regulations have been established worldwide to enhance construction safety and prevent accidents (Fang *et al.*, 2019). The developed safety policies can minimize the alarming statistics mentioned earlier by implementing best safety practices at construction sites.

Construction personnel must understand and access various best practices and safety rules to manage safety and health at the job site (Khan *et al.*, 2019). However, getting knowledge of everything, finding the proper safety rules, and then manually implementing them is tedious and not practical (Park, Lee and Khan, 2020). Thus, construction industry professionals have automatically devoted significant attention to monitoring safety rule compliance using vision intelligence-based approaches. However, the previous efforts to strengthen the safety rules monitoring in the construction job site are limited to specific hazards and not yet mature to apply for life-threatening situations. Also, controlling the entire job site using a single source simultaneously is not practical due to the huge number of participants in a large area and dynamic nature of construction site. Therefore, a GTM based analysis technique is used to develop a structured classification of safety rules for a compact vision intelligence-based system development for construction safety. The Occupational Safety and Health Administration (OSHA) rules are selected as a domain scope of this research. An error and trial-based criteria has been established by authors and experts to extract relevant safety rules from OSHA database. The safety rules are coded using open coding, axial coding and selective codes advised in GTM. The classified into four groups based on the open coding, axial coding, and selective coding approach: (1) Before Work, (2) With Intervals, (3) After Work (4) During Work. This structured classification are then linked with the scene capturing sources such as single scene capturing through smart phones for the rules required before work start and after work finish, periodic scene capturing using robots and drones for the rules required with intervals, and CCTVs for the safety rules required for continuous monitoring. The proposed classification framework is expected to pave the road for the prospect researchers and practitioners adopting vision intelligence-based monitoring systems, promote the bottom-up reporting approach, and enable the relevant safety rules compliance checking at right time.

Computer vision has attracted substantial attention because of the progress made in specific associated parameters with this domain, such as advances in high-definition cameras, convenient accessibility of the internet with excellent speed, developments in augmented storage for databases. As a result, computer vision-based methods have become prevalent for productivity analysis(Roberts and Golparvar-Fard, 2019), project progress monitoring (Golparvar-Fard, Peña-Mora and Savarese, 2015), and safety monitoring (Ding *et al.*, 2018; Fang *et al.*, 2018, 2019; Mneymneh, Abbas and Khoury, 2018; Wang *et al.*, 2019a). Recent research in computer vision-based construction safety monitoring has focused on developing a simple inspection system to detect safety preventive measures. For instance, hard hats detection (Mneymneh, Abbas and Khoury, 2018), safety harness recognition for fall hazards (Fang *et al.*, 2018), proximity detection (Wang *et al.*, 2019b), unsafe behavior detection in traversing structural members such as beams to make shortcuts (Fang *et al.*, 2019). However, the computer vision approach is still rapidly growing due to its cost-effectiveness, ease, and reliability.

### **2. Applications of Construction Safety Regulations**

Construction best practices and regulations provide lessons learned from the previous work (Khan *et al.*, 2019) and have a critical impact in establishing planning, design development, and work execution on job sites (He *et al.*, 2016). The construction task execution requires resources such as workers, materials, tools, and equipment. The interaction of these resources could cause hazards, and the best way might be the application of safety rules to control these hazards at the construction site (Zhang *et al.*, 2013). Even though construction industry professionals and government agencies have made the efforts, the current safety rule compliance still relies on manually auditing and supervising approaches that are inefficient and prone to error (Park, Lee and Khan, 2020). However, the recent trend of design for safety in Building Information Modelling (BIM) (Kasirossafar and Shahbodaghlou, 2013; Zhang *et al.*, 2013; Hongling *et al.*, 2016; Khan *et al.*, 2020), and computer vision-based safety monitoring (Fang *et al.*, 2018, 2019, 2020; Wu and Zhao, 2018) have attracted the interests of many researchers, however, yet in the elementary stages and need more attention for practical application in the construction industry.

#### **3. Need for Construction Safety Rules Classification**

There have been tremendous advancements in translating the natural language-based rules to make the machines understand binary languages (Kim *et al.*, 2019). However, due to the enormous number of safety regulations and the inherited complexities with them, finding and implementing the appropriate contents tends to be difficult (Hussain *et al.*, 2017). Previous computer vision-based safety monitoring efforts also revealed that vision-based research is limited to a few specific safety rule compliances. Thus, require comprehensive classification by investigating risk patterns to develop compact vision-based safety monitoring systems (Park, Lee and Khan, 2020). Moreover, controlling the overwhelming number of hazards in a huge job site needs multiple modes and image logistic devices for monitoring and controlling. These challenges require expert understanding and well-structured classification of a safety rule that can be acquired in computer vision-based safety monitoring. Therefore, this study focused on the OSHA regulations to validate the proposed concepts.

### **4. Methodology and Framework**

Establishing safety regulation classification criteria and mechanisms for the real-time provision of rules to safety supervisors is an essential part of safety management.

Table 1: Ground Theory Methodology analysis style

### **5. Occupational Safety and Health Administration (OSHA) Rules Analysis**

Motivated by the rapid development of industries in the 1970s, OSHA was established to enhance construction health and safety in 1971. Since then, the rate of reported serious workplace injuries (1972-2009) has significantly dropped from 11 out of 100 workers to 3.6 out of 100 workers (OSHA, 2009). OSHA collected and analyzed many accident cases, thereby creating an expert knowledge database. Based on that data, substantial amendments have been made to alleviate the past policies for modern industry compliance. The OSHA regulations for construction are developed under section 107 of the contract work hours and safety standards act, comprise a separate section of 27 subparts (OSHA, 2020).

Table 2: Analysis of OSHA safety Regulations

The parsing technique has been used to extract the data from the OSHA website automatically. A total of 8970 safety standards has been extracted as raw data. The data was filtered and cleaned using the established criteria in step 4 of Table 1; after the screening process, 5484 safety rules have been finalized for further research. As some sub-parts of OSHA 29-CFR 1926 construction were under amendments such as sub-part Z and sub-part CC, the safety rules analyzed remained 3538. These cited rules were thoroughly reviewed using GTM, such as allocating open codes to each rule. The open codes are then connected using their relationship with each other to form axial codes. After that, the axial codes are further narrowed up to develop the selective code. Based on the selective codes, 15.34% were classified as general rules, 8.5% were categorized under the procurement phase, 0.35% of rules can be adopted in the pre-construction phase, and 52.65% of rules were grouped under the construction phase, as mentioned in Table 2. This 52.65% of the rules were further selected for the analysis of workstage-based rule classification.

Figure 1: Example of Open coding, axial coding, and selective Coding

In the second stage, the OSHA safety rules that were being grouped under the construction phase were further analysed for computer vision-based safety monitoring using work-stagebased classification. The same approach mentioned in the GTM, such as open codes, axial codes, and selective codes, was formulated for the work-stage-based classification. The selective codes are mentioned in Figure 1. The safety rules are further grouped in four classes for work stage-based classification using the relationship of selective codes such as before work having 32.95% share, 41.59% for during work, 8.05% for with intervals and 13.09% for after work.

Figure 2: Workstage based OSHA regulation classification conditions

### **6. Examples from OSHA Regulations for Work-Stage based Classification**

### **Before Work**

#### Case-1

 1926.352(b) "If the object to be welded, cut, or heated cannot be moved and if all the fire hazards cannot be removed, positive means shall be taken to confine the heat, sparks, and slag, and to protect the immovable fire hazards from them." (See Figure 2.)

Figure 2: Example of Welding Activity for Safety Equipment Installation Before Work

## Case-2

1926.451(f)(3) Scaffolds and scaffold components shall be inspected for visible defects by a competent person before *each work shift*, and after any occurrence which could affect a scaffold's structural integrity. (See Figure 3.)

Figure 3: Example of scaffold components for Tools Inspection Before Work

## **With Intervals**

Case-1

1926.502(j)(6)(ii) "Excess mortar, broken or scattered masonry units, and all other materials and debris shall be kept clear from the work area by removal at *regular intervals*." (See Figure 4.)

Figure 4: Example of Broken or Scattered Masonry for Monitoring with Intervals

## Case-2

1926.1053(b)(15) "Ladders shall be inspected by a competent person for visible defects on a *periodic basis* and after any occurrence that could affect their safe use". (See Figure 5.)

Figure 5: Example of Ladder for Monitoring with Intervals

## **During Work**

Case-1

1926.451(c)(1)(iii) "Ties, guys, braces, or outriggers shall be used to prevent the tipping of supported scaffolds (such as mobile scaffold, etc) in all circumstances where an eccentric load, such as a cantilevered work platform, is applied or is transmitted to the scaffold." (See Figure 6.)

Figure 6: Example of mobile scaffold for Monitoring During the Whole Work

## Case-2

1926.451(f)(1) "Scaffolds and scaffold components shall not be loaded in excess of their maximum intended loads or rated capacities, whichever is less." (See Figure 7.)

Figure 7: Example of loaded Scaffolds for Monitoring During the Whole Work

## **After Work**

Case-1

"1926.706(b) All masonry walls over *eight feet in height* shall be adequately braced to prevent overturning and to prevent collapse unless the wall is adequately supported so that it will not overturn or collapse. The bracing shall remain in place until permanent supporting elements of the structure are in place." (See Figure 8.)

Figure 8: Example of masonry walls over eight feet in height for Monitoring after the Work Finish certain Level

### Case-2

1926.701(b) Reinforcing steel. All protruding reinforcing steel, onto and into which employees could fall, shall be guarded to eliminate the hazard of impalement." (See Figure 9.)

Figure 9: Example of protruding reinforcing steel for Monitoring Work Finish of Certain Level

### **7. Conclusion**

This paper proposes the classification of OSHA safety rules to develop a compact vision intelligence-based system using ground theory methodology. The rules are classified into two layers. First, the rules are classified in four groups; rules related to (1) procurements phase (8.5%), (2) pre-construction phase (0.35%), (3) construction phase (52.65%), and (4) general rules (15.34%). Second, the construction phase-related rules are further classified for work stage-related rules to adopt computer vision-based safety monitoring. The safety rules are further grouped into four classes such as Before Work, With Intervals, During Work and After Work having 32.95%, 8.05%, 41.59%, and 13.09% of share, respectively. In the extension work, image data capture devices will be compared against the identified classes of work stages.

### **Acknowledgment**

This work is supported by National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2020R1A4A4078916).

#### **References**

BLS (2017) National Census of Fatal Occupational Injuries in 2017, Bureau of Labor Statistics US Department of Labor.

BLS (2019) NATIONAL CENSUS OF FATAL OCCUPATIONAL INJURIES IN 2018. Available at: www.bls.gov/iif (Accessed: 29 January 2020).

Ding, L. et al. (2018) 'A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory', Automation in Construction, 86(March 2017), pp.118–124. doi: 10.1016/j.autcon.2017.11.002.

Fang, W. et al. (2018) 'Falls from heights: A computer vision-based approach for safety harness detection', Automation in Construction, 91(February), pp.53–61. doi: 10.1016/j.autcon.2018.02.018.

Fang, W. et al. (2019) 'A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network', Advanced Engineering Informatics, 39(December 2018), pp.170–177. doi: 10.1016/j.aei.2018.12.005.

Fang, W. et al. (2020) 'Computer vision applications in construction safety assurance', Automation in Construction. doi: 10.1016/j.autcon.2019.103013.

Golparvar-Fard, M., Peña-Mora, F. and Savarese, S. (2015) 'Automated Progress Monitoring Using Unordered Daily Construction Photographs and IFC-Based Building Information Models', Journal of Computing in Civil Engineering, 29(1), p. 04014025. doi: 10.1061/(ASCE)CP.1943-5487.0000205.

He, K. et al. (2016) 'Deep residual learning for image recognition', Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, pp.770– 778. doi: 10.1109/CVPR.2016.90.

Hongling, G. et al. (2016) 'BIM and Safety Rules Based Automated Identification of Unsafe Design Factors in Construction', Procedia Engineering, 164(June), pp.467–472. doi: 10.1016/j.proeng.2016.11.646.

Hussain, R. et al. (2017) 'Safety regulation classification system to support BIM based safety management', ISARC 2017 - Proceedings of the 34th International Symposium on Automation and Robotics in Construction, (Isarc).

Kasirossafar, M. and Shahbodaghlou, F. (2013) 'Building Information Modeling for Construction Safety Planning', Icsdec 2012, pp.1017–1024.

Khan, N. et al. (2019) 'Excavation Safety Modeling Approach Using BIM and VPL', Advances in Civil Engineering, 2019. doi: 10.1155/2019/1515808.

Khan, N. et al. (2020) 'Visual language-aided construction fire safety planning approach in building information modeling', Applied Sciences (Switzerland), 10(5). doi: 10.3390/app10051704.

Kim, H. et al. (2019) 'Visual language approach to representing KBimCode-based Korea building code sentences for automated rule checking', Journal of Computational Design and Engineering, 6(2), pp.143–148. doi: 10.1016/j.jcde.2018.08.002.

Mneymneh, B. E., Abbas, M. and Khoury, H. (2018) 'Vision-Based Framework for Intelligent Monitoring of Hardhat Wearing on Construction Sites'. doi: 10.1061/(ASCE)CP.1943-5487.0000813.

Nath, N. D., Behzadan, A. H. and Paal, S. G. (2020) 'Deep learning for site safety: Real-time detection of personal protective equipment', Automation in Construction, 112. doi: 10.1016/j.autcon.2020.103085.

OSHA (2009) Occupational Safety and Health Administration Timeline. Available at: https://www.osha.gov/osha40/timeline.html (Accessed: 15 March 2021).

OSHA (2020) OSHA 1926 Requirements Graphic Products. Available at:

https://www.graphicproducts.com/articles/osha-1926-requirements/ (Accessed: 15 March 2021).

Park, C., Lee, D. and Khan, N. (2020) 'An Analysis on Safety Risk Judgment Patterns Towards Computer Vision Based Construction Safety Management', p. 52. doi: 10.3311/CCC2020-052.

Roberts, D. and Golparvar-Fard, M. (2019) 'End-to-end vision-based detection, tracking and activity analysis of earthmoving equipment filmed at ground level', Automation in Construction, 105, p. 102811. doi: 10.1016/j.autcon.2019.04.006.

Wang, M. et al. (2019a) 'Predicting safety hazards among construction workers and equipment using computer vision and deep learning techniques', Proceedings of the 36th International Symposium on Automation and Robotics in Construction, ISARC 2019, (October), pp.399–406. doi: 10.22260/isarc2019/0054.

Wang, M. et al. (2019b) 'Predicting safety hazards among construction workers and equipment using computer vision and deep learning techniques', in Proceedings of the 36th International Symposium on Automation and Robotics in Construction, ISARC 2019. International Association for Automation and Robotics in Construction I.A.A.R.C), pp.399–406. doi: 10.22260/isarc2019/0054.

Wu, H. and Zhao, J. (2018) 'An intelligent vision-based approach for helmet identification for work safety', Computers in Industry, 100, pp.267–277. doi: 10.1016/j.compind.2018.03.037.

Zhang, S. et al. (2013) 'Building Information Modeling (BIM) and Safety: Automatic Safety Checking of Construction Models and Schedules', Automation in Construction, 29, pp.183–195. doi: 10.1016/j.autcon.2012.05.006.

## **Concept to support the estimation of static load capacity on construction sites using in-situ AR-based methods**

Christian-Dominik Thiele, Tim-Jonathan Huyeng, Pascal Mosler, Uwe Rüppel TU Darmstadt, Germany<EMAIL_ADDRESS>
**Abstract.** Due to the advancing digitization in the AEC industry (Architecture, Engineering and Construction), existing workflows are undergoing a transformation towards modern work patterns. In particular, there is potential for optimization in the area of building in existing structures. For example, the on-site capture of a comprehensive model in the context of Historic Building Information Modeling (HBIM) and especially of building elements can be supported by modern technologies. This paper uses an exemplary use case to demonstrate how such transformation from a traditional to a modern, digitally enhanced workflow is made possible to support an engineer onsite. The use case is the evaluation of a construction state with a temporary local load increase. The traditional workflow consists of multiple manual and partially iterative steps. The proposed workflow uses advanced augmented reality (AR) technology on mobile devices as well as a selfdeveloped Web API for an existing structural analysis software. This enables an on-site estimation of the static load capacity of a structural subsystem.

#### **1. Introduction**

Due to the aging real estate stock, especially of the public sector, more and more (infrastructure) buildings reach the end of their planned lifespan (Bigalke et al., 2016). There are no adequate capacities to assess the condition of such buildings and, if necessary, to determine their remaining lifespan. In addition, there is often a desire to maintain and repurpose buildings worthy of preservation rather than demolishing them. In case of protected existing structures and listed buildings, there is no other choice because demolition is out of the question. For the preservation or repurpose of the buildings, BIM models and plans could help in the assessment of the current building state. But these are in the case of historic buildings usually missing or incomplete. In order to apply digital methods such as BIM or HBIM to buildings of this kind and to analyze the current state, a complex stocktaking process is required. This is carried out by means of laser scanning in order to generate a point cloud from which a BIM model can be created. For the scanning, however, it is necessary to remove old pipes, non-load-bearing walls, other shoring and furnishings from the existing building.

For large construction sites in existing structures, there are also concerns about logistics such as material storage. With multiple stages of construction, building materials such as masonry units must be temporarily stored in hallways or rooms. Here, it is often unclear whether the ceiling structures can withstand the point loads. Since the construction progress often is timesensitive, there is a high interest to make a quick assessment of the static temporary loading capacity. This issue is fundamentally different from BIM-based planning. The structural engineer cannot perform the recalculations on the basis of a model or several plans, as it is usually the case with new buildings, but is dependent on on-site measurements. Due to the lack of a BIM model, the engineer will have to rely on a digital structure model they create for themself in order to perform the necessary calculations.

The aim of this paper is to digitally transform the presented processes, especially in structural planning, with modern in-situ methods like AR and the use of microservices. It is shown that a digital recording of the structure with instant calculation is possible and supports the engineer assessing the situation on-site. This approach has relevance far beyond construction sites and is also applicable, for example, for an immediate assessment of damaged buildings after disasters in order to release operations, e.g., for rescue forces. The focus here is to gain an immediate first impression of the situation, which can be used as a basis for a verifiable structural analysis.

## **2. Related work**

The term HBIM is primarily associated with the preservation of the architectural legacy of historic buildings. In order to generate a HBIM model, the structure is first captured for example by using a laser scanner. This results in a point cloud of the structure. Subsequently, the point cloud can be transformed into 3D component elements. In addition to the captured geometry, information about the material and construction type can be linked to the building elements, adding "intelligence" to the model (Murphy et al., 2013). This whole process is challenging due to the complex structure of historic buildings (Barazzetti and Banfi, 2017).

As described in Barazzetti and Banfi (2017), there is growing interest in developing AR and virtual reality (VR) applications for historic buildings. The authors present several exemplary implementations of HBIM models in smartphone or tablet applications. AR and VR applications are also being developed for specialized engineering applications in an attempt to move the workplace from the office to the construction site. One example mentioned by Barazzetti and Banfi (2017) is Autodesk 360 which highlights the benefits of AR applications. It promises great potential when combined with a BIM model, as relevant information can be obtained on-site from the BIM model, increasing work productivity.

Few professional structural analysis software such as RFEM (Dlubal, 2021) and SOFiSTiK (SOFiSTiK, 2021) have possibilities for interactions that are driven locally by scripts. RFEM, for example, can be accessed with various programming languages using the SDKs published by the manufacturer (Dlubal, 2015). In contrast, SOFiSTiK can be started in headless mode via the command line. Furthermore, the provided interfaces allow the readout and subsequent processing of calculation results. All structure analysis software of this type known to the authors do not have a documented Web Application Programming Interface (API). An exception is SkyCiv, a commercial cloud engineering software (Carigliano, 2020). It offers a (paid) server-side Web API. In addition to the mentioned software, there are freely available packages for various programming languages ("GitHub Finite Element Analysis", 2021).

Kudela et al. (2020) showed the possibilities of a static assessment of historic buildings by means of point clouds. Their approach is based on photogrammetry methods and the finite cell method for determining statically critical areas. The use of augmented reality on construction sites has been studied and discussed by various authors. Shin and Dunston (2009) presented a self-developed "AR prototype system for inspection" to check the placement and alignment of steel columns by distance measurements. Through series of experiments under ideal conditions, they found that the measurement accuracy was inferior to that of a total station but was sufficient for an initial assessment. They concluded that the use on a construction site requires greater robustness and more stable tracking technology.

Zhou et al. (2017) proposed a method for the position control of segments in tunneling. In this approach, marker-based calibration measurements are supposed to be aligned with a deposited BIM model. They found out that the measurements at larger distances are becoming inaccurate. This makes it impossible to reliably detect displacements with millimeter precision. They also pointed out that the use of markers on (tunnel) construction sites is impractical and therefore recommended a system without markers for comparable distance measurements.

Li et al. (2017) proposed a smartphone-based client-server system for finite element analysis at construction sites. The construction models are deposited on a server beforehand and are passed to the user after entering geometric or technical parameters. The smartphone application reads the data and virtually places the model in the environment using image tracking. Li et al. (2017) recommended applying finite element analysis also to dynamic models and to use a more reliable tracking technology.

Park et al. (2013) considered AR in interaction with ontologies for documenting damages in structures using smartphones and tablets ("AR-based Defect Inspection System"). The authors focused on the acquisition and processing of data and used markers for this purpose, as did Zhou et al. (2017).

### **3. Analysis**

The methodology of (H)BIM already enables a detailed description of a building. However, the preparation of the respective model is costly, especially for historical buildings. This is due to the complex structure and shape of irregular building components such as shell and arched structures. In addition, areas that are difficult to access, for example where walls or pipes have been subsequently installed, require time-consuming post-processing of laser scans. Therefore, unless simple structural systems are assumed, it should be estimated which level of detail provides the best value. Numerous researchers such as Barazzetti and Banfi (2017) are using the capabilities of AR and VR to visualize historic buildings in the context of HBIM. Gamification features are used to provide a credible simulation to interested users (storytelling and museum applications). According to Barazzetti and Banfi (2017), AR and VR applications are also increasingly used in specialized applications.

To the best of our knowledge, there is currently no standalone structural analysis software with a Web API available. However, it is possible to implement a Web API as a wrapper on top of the actual, already existing functions. If a structural analysis software is extended by a Web API, on-site calculations are thereby possible. Since using common standalone and established software is used, the on-site recording and the results can be also used for later analysis and can be easily integrated into existing workflows. The integration of the results or input files from non-standard programs into the existing workflow is in contrast much more complex.

The approach presented by Kudela et al. (2020) for the assessment of structures is particularly applicable for exposed and easy-to-reach structures such as bridge structures. However, a quick on-site assessment is not possible due to the complex image processing and subsequent calculation. Furthermore, the focus of the approach is on the detection of weak points of an existing system without considering possible additional loads.

The investigations of Li et al. (2017), Shin and Dunston (2009) and Zhou et al. (2017) have in common that all authors consider AR to be fundamentally suitable for the use on construction sites. In particular, the technical hurdles that were still apparent in the work of Shin and Dunston (2009) such as missing standardized AR-capable hardware and tracking technology were no longer present almost ten years later. The accuracy required for construction sites depends heavily on the use case. Distance measurements using AR currently run up against technical limitations in the millimeter range (Zhou et al., 2017), but are fundamentally faster and less complicated than using a total station. Regarding tracking technology, a trade-off between accuracy and robustness can be observed. Marker-based tracking appears unsuitable for the use on construction sites for logistical reasons (Zhou et al., 2017). Moreover, image tracking technologies such as Vuforia tend to be unreliable (Li et al., 2017). The regarded workflows foresee multiple people involved. Even though immediate results are instantly available after the measurements, like in Li et al. (2017), preliminary work has to be done before the actual use on the construction site. This includes modelling the FEM models and calibrating the tracking technology.

#### **Implications for further procedure**

Based on the approaches and findings of the related work, this paper aims to make the handling of the required technology fast and easy. The aim is to avoid any additional personnel and any preparatory work. This seems expedient since distance measurements on vaulted ceilings do not demand an increased accuracy. This will result in time and cost advantages. To further lower the technical hurdles, the use of AR-enabled mobile devices is preferred. VR technologies are not being considered because of the safety risks that would arise when they are used solely on a construction site. Instead, users must always keep an eye on their surrounding, which is only possible using AR. According to Mekni and Lemieux (2014), the three main characteristics of AR are 1) overlaying reality with virtual objects, 2) real-time interaction, and 3) threedimensional sensing.

In the next chapter, the status quo of the estimation of the static load capacity of a ceiling construction is described. The procedure is illustrated using the example of a common ceiling type in Germany, the Prussian vaulted ceiling.

#### **4. Analysis of the current workflow on the example of a Prussian vaulted ceiling**

A slab section constructed as a Prussian vaulted ceiling (as pictured in Figure 1) of a historical building is chosen as an example structure. The choice for this type of construction was made due to the easily visible structure as well as the simple static calculability. However, the discussed workflow is also suitable for other exposed structural systems such as wood beam ceilings. Vaulted ceilings are a historic, widely used construction method consisting of longitudinal beams with intermediate stone vaults. This construction method can be found in numerous old buildings towards the end of the 19th century in Germany (Fischer, 2009). In addition to residential and office buildings, this type of construction is present in some of Berlin's subway stations, such as the Sophie-Charlotte-Platz (Bezirksamt Charlottenburg-Wilmersdorf, 2014).

Figure 1: Cross-section of a typical Prussian vaulted ceiling with additional masonry pallet

### **4.1 Underlying example structural system**

In the chosen example (Figure 2), the longitudinal beams rest on the outer wall of the building on one side and are supported by steel girders in the field. This arrangement is typical for buildings with vaulted ceilings. The context is a temporary punctual load increase, e.g., due to a required storage area for building materials on a ceiling, such as masonry blocks. Therefore, the aim is to determine whether the present structural subsystem can handle such loads.

For a significant estimation of the static load capacity of a subsystem, the adjacent technical boundary conditions must always be considered and estimated correctly. Thus, when a partial structure is cut free, the effects from the outside on the subsystem (e.g. supports from floors above that rest on the system) as well as from the modified subsystem on the global system (such as foundations and walls) must be taken into account.

Due to the complexity that quickly arises when calculating an entire system, it can be argued that, for example, the redistribution of loads is not expected to have any effect on the entire system. This would mean that only local subsystems need to be verified. A typical example is a state of construction where temporary load increases are expected in the area of storage areas for construction materials. In this case, single additional loads in the construction state can often be compensated in the global system by omitted live loads from levels above or by missing floor superstructures. Thus, only the considered subsystem has to be examined for the local increased force. The load effect on the underlying walls or foundations mostly remains the same in total and does not have to be verified anew. The responsible engineer must assess whether this is the case. If so, it is possible to consider subsystems isolated in existing buildings and therefore reduce the effort required for the static proof of the structural condition.

For the selected use case, only the steel structure is considered. For an entire structural verification of the subsystem, it is still necessary to verify the static proof of all necessary components of the subsystem as the allowed pressure of the vaulting stones and connected structures. The described example structure of a Prussian vaulted ceiling has been modelled in SOFiSTiK (SOFiSTiK, 2021) to visualize the subsystem (Figure 2).

Figure 2: Left: *Alte Brauerei Meerbusch* (source: DEUTSCHE ROCKWOOL), Right: corresponding structural system, modelled in SOFiSTiK

### **4.2 Current workflow**

In order to prove a state of construction, a procedure as it is shown in Figure 3 is common. This process can become time-consuming because it can involve multiple trips to the site. Once the structure and the related subsystem are identified on-site, the recording of the surrounding area can begin. In the presented example of the vaulted ceiling, this would include measuring the length and the cross-section of the longitudinal girders as well as the spacing between two girders. For the purpose of identifying the girder dimensions, the visible flange width can be measured and potentially inferred to technical registers if the year of construction and girder type are known, e.g. from Bargmann (2013). Alternatively, more complex investigations can be necessary in order to get the dimensions of the girder. When columns are present, their length and related girder cross-section must also be measured.

The recording of structural systems on-site is conventionally done by using a folding rule or laser distance meter and handwritten notes and sketches. Depending on the situation on-site, a ladder or scaffolding may be required for recording. Furthermore, the acting loads are estimated. Once the recording is complete, it must be digitally modeled and then calculated using a FEM program, depending on the complexity of the system. When it becomes evident during modeling that values are missing or appear implausible, a new time-consuming recording on the construction site may be necessary. Once the calculation and documentation has been carried out, the intended state of construction can be approved or rejected. The final approval process can vary depending on the local regulations and standards.

Figure 3: Conventional workflow: recording and calculating a state of construction in terms of structural analysis

Since this process can be time-consuming and can require numerous steps, the authors propose an optimized workflow.

### **5. Optimization of the traditional workflow**

In the following, an approach for an in-situ evaluation of simple static systems by means of AR technologies is proposed. In addition, a Web API is suggested for the connection to an exemplary structural analysis software. Therefore, a Web API has been developed and a mobile application has been designed. For this, the use case described in Chapter 4 has been considered: the structural engineer is called to the construction site and has to assess a structural condition (Chapter 5.1). For this purpose, the engineer uses an AR-based application on their mobile device. The underlying system architecture is described in Chapter 5.2 and the mobile application in Chapter 5.3.

### **5.1 Proposed workflow**

The traditional workflow described in Chapter 4 has been optimized for the vaulted ceiling use case. It is described below and shown in Figure 4.

The developed application enables the digital recording of the subsystem's geometry directly on-site. With the help of AR software development kits (SDKs) for mobile applications, it is possible to perform distance measurements of points and recognize depth information just by using the camera of a mobile device. A vivid example is a tabletop whose surface can be detected using feature points and measuring the dimensions directly in the application. Once the geometry is captured, as described in Chapter 4.2, loads can be applied, cross-sections can be assigned to the line elements and boundary conditions of the system (such as supports) can be defined via the application. The geometric and semantic information will then be sent to the Web API, where the system is automatically calculated by the calculation engine of the structural analysis software SOFiSTiK. Immediately afterwards, the user receives the results of the calculation in the app. Finally, the engineer can check them for plausibility by validating the FEM analysis results directly on-site based on the received plots (like internal forces and bending lines, see Chapter 5.3). This allows the engineer to approve or reject the intended state of construction.

Figure 4: Proposed workflow: efficiently record and calculate a state of construction in terms of structural analysis

To supplement the documentation of the calculation, photos can be taken on-site directly via the application during the measurement process. Additionally, the documentation tools of SOFiSTiK can be used. A later post-processing of the model (e.g. in the office) via the graphical interface of SSD (SOFiSTiK Structural Desktop) is also possible.

### **5.2 System architecture**

Since the chosen structural analysis software SOFiSTiK cannot be installed on mobile devices, the calculation must take place on a stationary computer. Using an established structural analysis software has several advantages over a self-made FEM approach. An expert software is more sophisticated and offers a variety of support, which is essential for the reliability of the calculation. Furthermore, the recorded geometry can be reused in SOFiSTiK's extensive repertoire of calculation options for a later, more comprehensive calculation. It is also possible to use the established integrated tools SOFiSTiK Graphic and SOFiSTiK Report Browser for a concluding documentation. However, since SOFiSTiK is not designed for mobile use, the software does not provide a Web API. Therefore, a separate server application was designed as an interface for the chosen software.

With the help of the web framework FastAPI, a web service was implemented. It can be reached via HTTP(S) requests and represents the web interface to the structural analysis software. The relevant information for SOFiSTiK can be inserted either graphically or in a text-based format. Since the process flow shall be automated, the text-based format (DAT file) was chosen. The DAT file contains all relevant input data on the geometry of the structure. This includes loads, load cases and project information such as the standard on which the calculation is based. The creation of the input file is realized inside the mobile application. As Huyeng et al. (2020) suggested, it is also possible to use an external service for this purpose. SOFiSTiK consists of numerous modules, such as ASE (Advanced Solution Engine), the calculation kernel and the relevant module for the structural analysis itself. ASE can be started directly via the server application by means of a command line call and processes the transferred input file (DAT file). Therefore, no additional action from the user is necessary to perform the calculation.

Once the DAT file is created, it is forwarded to the so-called Sofi-Service which was developed as part of the *SCOPE* (Semantic Construction Project Engineering) research project (Huyeng et al., 2021; "SCOPE" 2018). After computation, the results are stored in a proprietary database (CDBase or CDB). This database can be accessed using program libraries published by SOFiSTiK. The Sofi-Service extracts the relevant parameters and calculation results, such as nodal results and displacements, and sends them back to the mobile application (see Figure 5).

Figure 5: Proposed system architecture and process flow

For documentation and optional later processing, the DAT file can additionally be sent back or stored and made available via a download link. This makes it possible to continue later and post-process the recorded structural system in the graphical user interface of SOFiSTiK (SOFiSTiK Structural Desktop).

## **5.3 Mobile application**

Figure 6: Mock-up of proposed mobile application using the example of *Alte Brauerei Meerbusch* (based on the photo of DEUTSCHE ROCKWOOL)

The proposed application should enable the user to measure the girder dimensions and distances by using built-in functionalities of the AR SDKs as described in Chapter 5.1. To mark the structural elements, the feature of drawing the structural lines onto the captured image of the mobile device camera needs to be provided. Additionally, the material (like HEB steel girders) needs to be assignable. Loads shall be added to the system by clicking on the previous created structural lines. In the same way, hinges and other mechanical parameters can be added to the system, too. Further settings for global parameters like the calculation standard and units can assist the workflow even more. After capturing the geometry and adding loads, the information will be sent to the server where the calculation will be carried out as described in Chapter 5.2. The calculation results can be visualized in different ways. Colored lines or the graphical representation of the deformation of structural elements are conceivable. As described in Chapter 5.1, the responsible engineer is expected to validate the results and to confirm the check for plausibility within the application.

#### **6. Conclusion**

In this work, we showed that AR tools integrated in a mobile application can optimize the way structural engineers work on construction sites. This applies in particular to buildings in existing structures. The field of applying mobile application is versatile and offers a high usability. In addition, the described approach supports ecological sustainable planning since analog drawings can be replaced and trips to the construction site can be significantly reduced. Since common AR SDKs are already supported by most common mobile devices, there is no need to purchase special expensive devices. According to Révész (2020), slightly more than a quarter of all smartphones in use are currently AR-enabled. Furthermore, by implementing a Web API, it was possible to connect complex structural analysis software to a mobile application, although it was originally designed as a desktop application. The use of Web APIs implemented as microservices also enables flexibility in the implementation of additional features.

The presented approach demonstrates how the redefined workflow can help to support structural engineers on the basis of already existing equipment (smartphones and tablets) and innovative tools (ARCore). However, it must be made clear that the approach should support engineers rather than replacing their work on-site. In order to prevent the risk of giving up responsibility to the application, the results from the automated calculation workflow always need to be checked for plausibility.

Conceivable further developments could be, as an example, an automated structure recognition, with which bearing elements such as columns and beams can be detected by an image recognition algorithm. In addition to the structural element lines, the whole point cloud of the room could be captured, and an HBIM model can be created and linked to the structural system. This could be realized by using already available techniques of photogrammetry. To increase the user experience and calculation accuracy, the app could be extended by implementing checks for calculability in the background while the user enters the structural model. These checks could consider e.g. the statical determinacy.

### **References**

Barazzetti, L., Banfi, F., 2017. Historic BIM for Mobile VR/AR Applications, in: Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, G. (Eds.), Mixed Reality and Gamification for Cultural Heritage. Springer International Publishing, Cham, pp.271–290.

Bargmann, H., 2013. Historische Bautabellen : Normen und Konstruktionshinweise 1870–1960, 5. Aufl. ed. Köln.

Bezirksamt Charlottenburg-Wilmersdorf, 2014. URL https://www.berlin.de/ba-charlottenburgwilmersdorf/ueber-den-bezirk/gebaeude-und-anlagen/bahnhoefe/artikel.159800.php (accessed March 2021).

Bigalke, U., Armbruster, A., Lukas, F., Krieger, O., Schuch, C., Kunde, J., 2016. Statistiken und Analysen zur Energeieffizienz im Gebäudebestand, dena-GEBÄUDEREPORT. Deutsche Energie-Agentur GmbH (dena), Berlin.

Carigliano, S., 2020. Strukturanalyse- und Entwurfs-API | SkyCiv. SkyCiv Cloud-Strukturanalyse-Software | Cloud Structural Analysis Software and Calculators. URL https://skyciv.com/structuralanalysis-design-api/ (accessed October 2020).

Dlubal, 2021. RFEM [WWW Document]. FEM-Statiksoftware RFEM | Dlubal Software. URL https://www.dlubal.com/de/produkte/fem-statik-software-rfem/was-ist-rfem (accessed March 2021).

Dlubal, 2015. RFEM-/RSTAB-Zusatzmodul RF-COM/RS-COM [WWW Document]. Dlubal. URL https://www.dlubal.com/de/produkte/rfem-und-rstab-zusatzmodule/sonstige/rf-com (accessed February 2021).

Fischer, M., 2009. Steineisendecken im Deutschen Reich 1892–1925. BTU Cottbus - Senftenberg.

GitHub Finite Element Analysis [WWW Document], 2021. Topic: Finite Element Analysis. URL https://github.com/topics/finite-element-analysis (accessed March 2021).

Huyeng, T.-J., Thiele, C.-D., Rüppel, U., Sprenger, W., 2020. An approach to process geometric and semantic information as open graph-based description using a microservice architecture on the example of structural data. Presented at the European Group for Intelligent Computing in Engineering, Berlin.

Huyeng, T.-J., Thiele, C.-D., Wagner, A., Shi, M., Hoffmann, A., Rüppel, U., Sprenger, W., 2021. Interlinking geometric and semantic information for an automated structural analysis of buildings using semantic web. Presented at the European Conference on Product and Process Modelling, Moscow, Russia, p. to appear.

Kudela, L., Kollmannsberger, S., Almac, U., Rank, E., 2020. Direct structural analysis of domains defined by point clouds. Computer Methods in Applied Mechanics and Engineering 358, 112581.

Li, W.K., Nee, A.Y.C., Ong, S.K., 2017. Mobile augmented reality visualization and collaboration techniques for on-site finite element structural analysis. International Journal of Modeling, Simulation, and Scientific Computing 09, 1840001.

Mekni, M., Lemieux, A., 2014. Augmented Reality: Applications, Challenges and Future Trends 10.

Murphy, M., McGovern, E., Pavia, S., 2013. Historic Building Information Modelling – Adding intelligence to laser and image based surveys of European classical architecture. ISPRS Journal of Photogrammetry and Remote Sensing, Terrestrial 3D modelling 76, 89–102.

Park, C.-S., Lee, D.-Y., Kwon, O.-S., Wang, X., 2013. A framework for proactive construction defect management using BIM, augmented reality and ontology-based data collection template. Automation in Construction, Augmented Reality in Architecture, Engineering, and Construction 33, 61–71.

Révész, R., 2020. How many AR enabled phones are in the world? AR-ON Platform. URL https://www.aronplatform.com/mobile-ar-penetration/ (accessed March 2021).

SCOPE – Semantic Construction Project Engineering, 2018. URL https://www.projekt-scope.de/ (accessed February 2021).

Shin, D.H., Dunston, P.S., 2009. Evaluation of Augmented Reality in steel column inspection. Automation in Construction 18, 118–129.

SOFiSTiK, 2021. sofistik.de [WWW Document]. FEM, BIM und CAD Software für Bauingenieure | SOFiSTiK AG. URL https://www.sofistik.de/ (accessed March 2021).

Westfeld, P., Mader, D., Maas, H.-G., 2015. Generation of TIR-attributed 3D Point Clouds from UAV-based Thermal Imagery. Photogrammetrie - Fernerkundung - Geoinformation 2015, 381–393.

Zhou, Y., Luo, H., Yang, Y., 2017. Implementation of augmented reality for segment displacement inspection during tunneling construction. Automation in Construction 82, 112–121.

## **Using eye-tracking to compare the experienced safety supervisors and novice in identifying job site hazards under a VR environment**

Yewei Ouyang, Xiaowei Luo City University of Hong Kong, Hong Kong<EMAIL_ADDRESS>
**Abstract.** Hazard-identification experience is a kind of tacit knowledge which is difficult to be extracted from experienced subjects and to be described explicitly in the text. Researchers have applied eye-tracking technology in eliciting the cognitive processes of experienced workers while performing the hazard-identification task. However, the image-based tasks in previous studies are substantially different from how the hazards are perceived on the construction site. To improve the ecological validity of the hazard-identification task, this study develops panoramic VR scenarios of various job sites as the stimulus, and both experienced safety supervisors and students are invited to identify hazards in the virtual sites. Their performances and eye-movement data are compared. The results show the experienced allocate more attention to hazardous areas instead of unimportant things, and they inspect more details which are ignored by the novice. The identified differences may be incorporated into the training courses to educate the hazard-identification of the novice.

#### **1. Introduction**

The construction site involves highly dangerous work and harsh work environment, so it is essential to train subjects' hazard-identification ability to avoid them from being hurt. Since hazard-identification is a complex task that requires knowledge of both regulations and experience because of the dynamic nature of construction environments, it might be useful to understand how experts search for and identify hazards and extract their experience to formulate explicit strategies that can be included in training materials. However, the hazard-identification experience is a kind of tacit knowledge which is difficult to be extracted from experienced subjects and to be described explicitly in the text.

Researchers have found the eye-tracking technology which measures eye position and movement, provides eye-tracking information such as fixation location and duration which can indicate one's cognitive strategies and prior knowledge or experience (Hyönä et al., 2002). Hence, researchers tried to utilize eye-tracking in the construction field in eliciting the cognitive processes of experienced workers while performing the hazard-identification task (Dzeng et al., 2016), and they compared the differences among workers with different years of working experiences (Hasanzadeh et al., 2017). Those previous studies used static images as the stimulus, but the image-based tasks are substantially different from how the hazards are perceived on construction site because static images fail to portrait dynamic job sites and the twodimensional images would lead to information shrinkage and changes in the cognitive process (Sun and Liao, 2019).

To improve the ecological validity of the hazard-identification task, the authors intend to apply virtual job sites created by panoramas of real sites as the stimulus. The panoramic VR creates highly realistic and detailed representations of real construction sites while giving users a sense of immersion(Moore et al., 2019). In such an environment, subjects are allowed to observe their surroundings to identify safety hazards exactly as what they do in real life. The virtual scenes are expected to have more conformities with real-time than the image-based task. Besides, it costs less time and represents details of job sites better than the virtual scenes created by 3D modelling, and compared with the real environment, it allows repeatable experimental conditions, which provides the same conditions for all participants. Besides, due to the simpleto-capture of panoramic pictures, a wide range of job sites covering various construction situations and safety hazards can be captured to acquire a more comprehensive comparison.

This study will integrate panoramic VR technology with eye-tracking technology, where the panoramic VR will be used to develop virtual job sites for subjects to search for hazards, and eye-tracking will be used to indicate subjects' safety experiences. The objective of this study is to find out the differences between the methods employed by experienced safety supervisors and novice of how they identify hazards in the panoramic VR scenes, which can further be incorporated into the training of hazard-identification for the novice.

## **2. Related Work**

Since eye-tracking can provide fixation information to indicate subjects' cognitive strategies and prior knowledge or experience, several studies have successfully applied eye-tracking methodologies to evaluate the difference between the methods used by the experienced and the novice. In the domain of traffic safety, Hosking et al. (2010) evaluated the difference between the visual search patterns that experienced and inexperienced motorcycle riders employ to identify road hazards; Pradhan et al. (2005) compared the scanning behaviour of experienced drivers and novice drivers under risky driving conditions, and they found novice drivers typically look directly ahead and fail to perceive and assess hazard information. These studies from other disciplines facilitate such methods to be applied in the construction industry. Since eye-tracking information can assist in eliciting the cognitive processes of experienced inspectors while performing search tasks (Sadasivan et al., 2005), researchers in the construction field used eye-tracker study the effect of experiences in hazards-identification. Dzeng et al. (2016) invited experienced and novice workers to identify hazards in four screenshots developed by Google SketchUp, which presents both obvious and unobvious hazards of workplaces. They used an eye-tracker to compare the differences in the workers' searching patterns for hazard identification. They reported the experienced workers were found to recognize hazards significantly faster than the novices, and their scanning paths are more consistent. Hasanzadeh et al. (2017) adopted 35 images of job sites as a stimulus to conduct an eye-tracking experiment. They found relative to less-experienced workers (<5 years), more experienced workers (>10 years) need less processing time and deploy more frequent short fixations on hazardous areas to maintain situational awareness of the environment.

However, how subjects perceive hazards on construction sites is substantially different from identifying hazards on images (Sun and Liao, 2019), and some elements of real job sites such as weather and noise are impossible to be expressed (Kushiro et al., 2017). Sun and Liao (2019) proposed to use a civil engineering laboratory on campus as the stimulus for an eye-tracking experiment exploring ability assessment of hazards-identification, they suggest the lab can be a simulated job site because the lab has consistent hazards with job sites and it has stationary site condition. Even though the lab has the same kinds of safety hazards as a job site such as fall-from-height or electric shock, its environment and contained objects are quite different from construction sites. The best choice is to adopt real job sites as the stimulus. Hasanzadeh et al. (2018) used mobile eye-tracking to measure workers' situation awareness towards tripping hazards in a live construction site, but they mentioned the site changes over extended amounts of time so that the research team had a short amount of time to test the subject. However, it is quite hard to invite a large number of participants to conduct such an experiment in job sites as the site condition is changing all the time.

While VR is a sufficiently authentic simulation of a construction site and it also allows repeatable experimental conditions compared with the real environment. Many studies tend to develop virtual scenarios to analyze subjects' behavior in real life, e.g. people's responses during indoor evacuation route-finding (Tian et al., 2019). However, developing VR scenarios using 3d models in game engines such as Unity 3D needs high computational costs, long development times, and has a limited sense of presence and realism (Moore et al., 2019). The panoramic VR would be a better choice. It applies 360-degree panoramas to create true-toreality simulations of environments and it also gives subjects a high sense of presence when presenting in a VR headset. Hence, this study proposes to use panoramic VR as the stimulus.

### **3. Research Methodology**

Firstly, panoramic VR scenarios representing job sites containing hazards are developed; Secondly, an eye-tracking experiment is conducted where participants are required to search for hazards in the panoramic VR scenarios; Finally, experimental data is analyzed to find out the differences between the experienced and the novice.

## **3.1 Panoramic VR Development**

Firstly, 18 panoramas are shot in job sites of building projects using a panorama camera (THETA Z1, RICOH), with the camera standing on a proper point which ensures the hazards can be seen clearly. Figure *1* shows two of the panoramas. Then the panoramas are added in Tobii Pro Lab as the stimulus, and they are presented to participants using HTC VIVE Pro Headset so that participants can feel immersed in the job sites.

The panoramic VR consists of the 18 job site scenes, covering 8 types of locations (i.e. pile foundation construction, foundation pit construction, construction of the main structure, construction of interior decoration, reinforcement yard, housekeeping area, office area and construction elevator) and 7 types of hazards (i.e. falling-from-height, struck-by objects, collapse, electric shock, improper housekeeping, explosion and improper personal protection). There are 172 hazards in total. Besides, these scenes are of various level of visual clutter. The authors adopt various site conditions aiming at achieving a comprehensive comparison for this study.

a) Indoor scenario b) Outside scenario

Figure 1: Two examples of the Panoramas

## **3.2 Eye-Tracking Experiment**

**Participants.** 20 undergraduate students in the last year majoring in construction management and 20 safety supervisors with over 3 years of experiences are invited to participate in the study. All participants have uncorrected normal vision or corrected-to-normal vision.

The students have learned all lessons related to construction safety, but they only once visited job sites for a short duration. The students are chosen as participants because they are the same as the novice who just graduate from university and are new to the job site.

The safety supervisors come from various project organizations with a multi-level of positions, including safety managers from the developer, chief safety directors and safety inspectors from both the contractor and sub-contractor and engineers in charge of safety from the supervisor. Their years of experience ranges from 20 to 3, with an average value of 8. Their past working experiences are inquired before the experiment to ensure they have enough experiences to identify the hazards.

**Experiment Process.** During the experiment, participants wear HTC VIVE Pro Eye (120 HZ), standing inside the playing area (i.e. the square area with the line between the two base stations as a diagonal) and moving around to observe the virtual job sites, as shown in Figure *2*.

Before the formal experiment, participants receive a briefing of the experiment and practice searching hazards in several sample scenes different from the later testing scenes, eliminating the impacts on hazard-identification performances caused by deficient mastery of the experiment requirements and inadaptation to the VR.

Calibration is done first to ensure the accuracy of eye-tracking measurements. Participants are instructed to fixate on five target points at different locations in their VR view. The calibration process establishes a mapping from pupil to gaze coordinates. After successful calibration, a sample scene appears again to ensure the participants can see their front view clearly.

When the testing scene appears, participants move around inside the play area to observe the panoramic job site. They are required to click the mouse in their hand whenever they discover any potential hazard, with their gaze stopping on the hazard. Each scene lasts 90 seconds and participants can switch to the next scene once they finish identification. The 18 scenes are presented randomly among all participants. There is a blank view without time limits designed between two scenes where participants can take a short rest before observing the next scene. The experimenter will direct the participant to the original orientation (participants face the experimenter at the beginning) so that all participants can see the same view when they first see a scene. The experimenter will also direct the participant to the centre of the play area if the participant goes to the edge of the area when observing the previous scene. Participants' mouseclick event and eye movement are recorded during the whole identification.

After the whole identification, the experimenter shows participants the views captured when they click the mouse. Participants are required to explain the hazards they point out. Their identifications will be regarded as correct only when their explanations are correct.

Finally, all participants are required to finish a questionnaire for collecting their personal information and their feedback about this experiment. In terms of personal information, students' major and job site experiences are collected, and supervisors' previous working experiences and their current work positions are collected. In terms of the feedback, both of them are required to describe their dizziness, tiredness and sense of reality towards the panoramic VR, the searching patterns they adopted, and their other feedback. The supervisors are also asked to talk about the use of experiences in hazard-identification.

Figure 2: Experimental Set-up

### **3.3 Data Collection and Analysis**

Hypothesis testing for a two-independent sample is used to examine whether there are significant differences in identification performances and eye-movement patterns between the experienced safety supervisors and the students. The identification performances are indicated by identification time and accuracy. The eye-movement patterns are indicated by fixationrelated eye-tracking metrics. Fixations are those states when an individual's eyes essentially stop scanning the scene, holding the foveal vision in place so that the visual system can take in detailed information about what is being looked at. Fixation is generally associated with attention, visual processing, and information absorption (Holmqvist et al., 2011). Fixation Counts referring to the number of fixations inside an area and Fixation Duration referring to the time subjects fixated on an area are chosen in this study to indicate participants' attention allocation.

Participants' identification accuracy and final feedback are recorded manually. Participants' identification time and eye-movement are collected by the software Tobii Pro Lab. The identification time of each scene ranges from the stimulus-start to the stimulus-end. To further compare the specific differences of eye-movement patterns, this study defines potential hazardous-areas as AOI (Area of Interest), as shown in Figure *3* (coloured areas are AOIs). In eye-tracking studies, AOI is an area in the display or visual environment that is of interest to the research. There are 6 kinds of AOIs defined, that is "person, scaffold, edge and hole, housekeeping, electric shock and other objects under unsafe status".

Figure 3: An example of AOI definition

#### **4. Results**

The results are presented from three levels: the result of the whole scenes (18 scenes), the result of each scene and the result of each AOI. At first, the following two conditions are examined to determine whether the independent-samples T-test can be used: 1) There are no outlier independent variables; 2) The dependent variables in each group obey normal distribution. Otherwise, the non-parametric test method (Mann - Whitney u test) will be used.

Table *1* shows the statistical results of the whole 18 scenes, which shows there are significant differences (*p < 0.05*) in identification time, identification accuracy, fixation counts and percentage of fixations, indicating students spend less time on identification, identify fewer hazards, have fewer fixations and spend less time in fixating.

Table *2* shows the statistical results of identification accuracy on each kind of hazard, that supervisors have significantly higher accuracy (*p < 0.05*) on all kinds of hazards. Students only identify some obvious hazards which can be seen under unsafe status from the appearances, e.g. workers not wearing hardcat, workers being too close to machines, materials stack blocking exits, unlocked electricity box, cables disorderly placed on the ground, holes on the ground without cover. Besides, none hazards caused by unsafe status of scaffold is detected by students.

Table 1: Statistical results of the whole 18 scenes

\**Percentage of fixations is the ratio between fixation duration and identification time, indicating how much time is spent in fixating.*

Table 2: Statistical results of identification accuracy on various hazard types

Table 3: Statistical results of AOI duration and its percentage

\* *Percentage of AOI duration* (*AOI\_duration%) = fixation time spent on AOI / total fixation time*

Table *3* shows the statistic result of AOI duration and its percentage, where AOI duration refers to the fixation time spent on AOI, indicating how much attention is allocated to a certain AOI; the percentage of AOI duration refers to the ratio between the fixation time spent on AOI and the total fixation time, indicating to what extent attention is allocated to a certain AOI. The differences are significant when calculating the whole AOIs. As for each kind of AOI, the results show supervisors significantly have longer fixation duration on all kinds of AOI except on the person. Similarly, supervisors have a significantly larger percentage of AOI duration on most AOIs except the housekeeping where no significant differences exist and the person where the students have a larger value. The statistical results of each scene are concluded in the last row of Table *1* and the last column of Table *3*, showing the number of scenes where significant differences exist.

Motion sickness occurs on VR players due to the disparity between the users' visual and vestibular stimuli (Clay et al., 2019), nevertheless, the panoramic job sites are static in this study, which greatly reduces participants' dizziness. As a result, none reported dizziness due to VR. One student reported dizziness due to the frequent movement of his body. None reported tiredness during the whole identification.

All students reported they feel going to the job sites personally. Merely one student reported the view resolution is a little lower. All supervisors reported the task is the same as their daily work except they cannot go far in the VR and there is scale distortion in some scenes which bring trouble in exactly determining distance or height.

#### **5. Discussion**

#### **5.1 Differences in Identification Time and Accuracy**

Table *1* shows supervisors have much higher accuracy (*p = 0.000*), which is not consistent with the study of Dzeng et al. (2016) which reported the experience do not help improve identification accuracy. Such inconformity might come from different experimental stimulus. Their total number (14 in total) and kinds of hazards are quite limited, which might be limited in detecting the differences between the two groups. The simple-to-capture of panoramic pictures allows a large number of safety hazards which helps to acquire a more comprehensive comparison. Besides, the students heavily rely on personal feelings to search for hazards because they lack related knowledge or experience to perceive hazards, so most of their identified hazards are those look obviously.

Even though supervisors spend more time on the whole identification (*p = 0.036*), their unit identification time ( identification time / identification accuracy) is significant less than students (*p = 0.000*), indicating they identify faster than students. This is consistent with what Dzeng et al. (2016) found that the experienced identify hazards faster than the novice.

## **5.2 Differences in Attention Allocation and Search Pattern**

Table *1* shows students have significantly fewer fixation counts than supervisors (*p = 0.004*), the authors further examined the Heat Map of fixation counts to analyze the specific differences. The Heat Map uses different colours to illustrate the number of fixations participants made within certain areas of the stimulus. Red indicates the highest number of fixations, and green the least, with varying levels in between. Figure *4* shows a pair of Heat Map. After observing the Heat Map of every scene, the authors found supervisors paid attention to more objects in the scenes, while students ignored some objects. Students' lack of attention to objects might a cause leading to their less identification time and lower accuracy.

When further looking at their attention allocation among the six kinds of AOIs, the specific differences in Table *3* are discussed below:

**Person.** Students are quite aware of persons in the scenes that they give a larger part of attention on persons than supervisors do (*p = 0.000*), as shown in Figure *4*a, humans are all red in the Heat Map of every scene, while there is no such pattern for supervisors. Nevertheless, students identify less unsafe acts. Supervisors' experiences drive them to inspect more details, that they inspect whether persons tie up their hardcat strap because they know workers not wearing the hardcat well is very common in daily work. Students also fail to identify the hazards requiring professional knowledge even though they notice it, e.g. workers do not place extinguisher when doing hot work.

Figure 4: One scene's Heat Map of fixation counts

**Scaffold.** Students hardly have awareness to check the safety status of scaffold, while supervisors are quite sensitive (*p = 0.000*). a) Students' Heat Map b) Supervisors' Heat Map

Figure *5* intuitively shows such differences, where students merely go through the scaffold and find no problems, supervisors exactly know the inspection points and they can correctly point out the hazards. In addition, supervisors even carefully inspect the scaffold on the buildings' external surface because they report there are always safety problems in real sites. Students report they once learned related knowledge in class, but they cannot remember and apply it. As a result, students do not identify any hazards caused by unsafe scaffold.

a) Students' Heat Map b) Supervisors' Heat Map

Figure 5: Heat Map of the scene with indoor scaffold

**Housekeeping.** Students spend less fixation duration in observing housekeeping (*p = 0.023*) even though they put considerable attention on it. The material-stack is less likely to be ignored due to its relatively large volume in the scene, and students are sensitive about its tidiness that they even mistake messy stack as hazards. Even though paying attention to material-stack, students fail to check whether there are extinguishers near flammable materials and to check whether the material-stack is over a safe height.

**Edge and Hole.** Students aware of the holes' cover on the ground, but they are less sensitive in examining edge protection than supervisors, e.g. they hardly look up at buildings' evaluation surface to ensure whether there are protections alongside the window opening, and they tend to ignore edges with elevation difference less than 2m.

**Electricity and Others.** Students regard electricity boxes as a source of danger and pay attention to them every time, but their experiences about electric hazards are limited that they fail to detect wherever there is no insulating bush around cables placed on mental objects like scaffold. Likewise, students almost noticed all other objects under unsafe status, but they only pointed out obvious hazards such as a plank is not fixed well on the door frame.