such as name, references, and revision, which was not present.

    * –

Inexperienced tester: Expected download links for the log files.

  * •

Task 6 - Specific step: Correct action available for the participants were
taken to achieve the task. Expected more information in the expected results,
including action description, expected result, actual result, and verdict.
Apart from that, the information and data in this view were satisfactory.

#####  5.3.2.2 Semi-structured Interview

The participants thought the prototype would be very useful for test analysis
and diagnostics and would lead to efficiency improvements. The participants
felt that, if the prototype could be used as a reporting tool for each test
case, where testers would be required to fill the prototype’s content with
data and information as a requirement, it would improve the accessibility and
shareability with other stakeholders involved.

Overall, the participants said that the prototype gave a good overview of the
test case if all the expected data and information was presented. By getting a
good overview right from the start, the test case’s context would be easier to
understand, which would lead to a more confident start to the test analysis
and diagnostics.

The Inexperienced participant appreciated this, as it could be challenging to
know what to look during the test analysis. The more experienced participant
found it easier to navigate the data and information than the current analysis
method.

According to the participants, the prototype’s most useful parts were the
historical view and information view. They required more effort to get this
information manually in the existing test analysis process.

#####  5.3.2.3 Improvements

The input gathered from the task analysis, and semi-structured interview were
as followed:

  * •

Overview view

    * –

The calibration should fit in the DVM view because the DVM view is responsible
for this information.

    * –

Date and time when the test case was conducted should be added.

  * •

DVM view

    * –

DVM should include pre-conditions, post-conditions, sensor signals, buses,
tools and description.

    * –

Test sequence should also include the actual results and verdict.

    * –

More detailed information of the requirement which the test case verifies
should be shown. Only DVM is not enough.

  * •

Historical view

    * –

Should also include the name and domain of the test case.

  * •

Information view

    * –

Should include software and hardware readouts such as ECU serial numbers and
software part numbers.

    * –

Should include measurement configuration for measurements tools.

  * •

Links

    * –

Have its own view which includes links and references to the requirements, DVM
and log files.

  * •

Specific step view

    * –

Have a summary of all steps.

    * –

Favorite signals was not useful, should be removed.

    * –

Historical test cases in this view were not useful, should be removed.

####  5.3.3 Evaluation Phase Findings

Having the data required to perform test analysis in one place was positively
perceived by participants in the evaluation phase. The prototype eases the
challenges they faced in the current method of gathering the data. The risk of
misinterpretation and accessing the wrong information was believed by the
participants to be reduced. The efficiency in beginning test analysis was
improved. Participants acknowledged that they had to manually find and locate
the old system’s required information and data in case another tester did not
provide them.

It can be very overwhelming for inexperienced testers to work in many
different tools to get the required data for test analysis. Most of the
inexperienced testers were not fully adapted to the testing and analysis
process. They admitted that it was common to get lost and ask what to look for
in the current tools. The more inexperienced tester perceived the prototype as
an excellent tool where the required information and data, which they needed
for test analysis, were already present. The more experienced testers thought
having the data in one tool made it easier to navigate between the pieces of
data, making it easier to look up things they were uncertain about.

By providing the required data that was necessary to start test analysis in a
structured manner improved the overall quality of information available during
test analysis by only showing relevant data and thus reducing
misinterpretation of information. Having only one tool instead of multiple
tools increased the efficiency to navigate between the test analysis data.
Such improvements reduced the risks to get overwhelmed and lost during
navigating large amount of data, by improving navigation efficiency during
test analysis.

###  5.4 Usability Test Results

The usability test results comprised time for completion of tasks and
calculations of the NASA Task Load Index with the Think Aloud protocol,
followed by interviews to get further insights into the usability aspects of
the system. This section presents the results and its analysis. Both the
existing system and the new system were compared during the usability test.
Testers performed four distinct tasks on both systems and their completion
time of the tasks along with the NASA TLX scores were recorded.

The usability test tasks are described in Section 4.5.1 where four tasks were
conducted sequentially by the participants. Each task was associated with
requirements of the New System and a mapping can be seen in Table 5.8. The
findings from the results of Task 1 and 4 relate to the first research
question, which targeted the aspects that deteriorated decision making during
test analysis and diagnostics of automotive electronics. In contrast, the
findings from Task 2 and 3 relate to the second research question, which
focused on data structuring, presentation and documentation to support
decision making and aiding testers to communicate test analysis and
diagnostics verdicts efficiently.

Table 5.8: Usability test task mapping to requirements. Task | Task Name | Requirement
---|---|---
Task 1 | Understanding the issue | UR1, UR8, UR9
Task 2 | Gathering information and data | UR2, UR6, UR10, UR11, UR12
Task 3 | Solving the issue | UR3, UR4, UR5, UR13, UR14, UR15
Task 4 | Reporting the issue | UR6, UR7

####  5.4.1 NASA TLX Results

According to the NASA Task Load Index guide, each task was measured based on
six subjective sub-scales, determining the workload index for each of them, as
explained in Section 4.5.4.1. Full results of the NASA Task Load Index for the
old and new systems can be found in in Appendix I and Appendix J. The mean
index score for each of the tasks can be seen in Figure 5.5. The scores are in
the range of 0-100 where lower scores refer to the less mental workload on
test subjects while performing tasks, which is better.

Task 1Task 2Task 3Task
4$0$$20$$40$$60$$80$$100$$46.5$$52.8$$48.8$$49.1$$22.5$$16.8$$37.5$$22.5$Mean
scoreOld SystemNew System Figure 5.5: NASA Task Load Index mean score of each
usability test task.

####  5.4.2 Think-Aloud Protocol

The think-aloud protocol used in the usability test explained the subjects’
behavior, which was posed as a supplement to the findings discussed in Section
5.4.3. During the performance of tasks, the testers found it easy and
intuitive to perform tasks 1, 2, and 4, especially when using the new system.
Despite the new interface, they could navigate using the new system’s logical
information structure as a guide. The same tasks, performed in the old system,
did not create an inspiring think-aloud environment as the participants were
very used to those tasks. There were glimpses of relief, especially when
performing Task 1, as they realized that information to complete the task was
all in one place in the new system.

For Task 3, as it required much thinking in itself, testers thought out loudly
more often, regardless of the system used. The more experienced subjects
talked about the need to use other measurement tools while working on the task
to complement the new system on more advanced test cases. They later realized
that for a test case with a more average difficulty, it wasn’t required at
all.

####  5.4.3 Interviews

The section explains the findings during the decoding phase of the interviews
(previously discussed in Section 4.5.6), done after performing the usability
test. A broad elaboration is given below along with a summary of findings in
Table 5.9 and full data in Appendix K.

The findings from the interview were generally positive for the new system.
The experienced and inexperienced subjects mostly reached a consensus
regarding the feedback. They favored the improved structure for storing the
files for the test cases and test objects in a primary domain. All the
subjects agreed that the better structure drastically improved the time
required to locate information and files.

"A lot of information and data in the old system is stored in files and how we
share with each other with the current tools, it can be very limiting and not
everything can be shared in a optimal way. I think as the new system contains
and displays all that, one can find them in faster and more efficient way."
\- Testers’ feedback.

The presentation of the data was improved over the old system. The subjects
could relate to the efficient categorization of the information into logical
pages in the user interface with faster information access. Everyone agreed
that the new system gave them a quicker overview of the data the subjects
required to perform test diagnosis. The subjects acknowledged that the
presence of all the data in one domain was a substantial improvement as it
would reduce mistakes due to misinformation and misinterpretation.
Traceability of data for test objects and test cases were also better
expressed due to logical structuring and presentation in a cleanly designed
user interface.

"Data and files for testing done personally/ad hoc basis are not managed in a
location where everyone can find it. Having such information in one place is
convenient in terms of getting information fast and reliably comparing the
results from the perspective of history."  \- Testers’ feedback on
traceability of testing artifacts.

Another vital piece of feedback was that the new system drastically reduced
the frequency of navigating other tools. The subjects stated that the saved
time would benefit them because they could conduct a test analysis on more
test cases in the same time period.

Regarding the usability of both the systems, they noted the ease of use of the
new system, which was particularly attractive for testers who were less
experienced. Overall, the realization was that the new system would provide
them with all the information needed to diagnose test cases problems, which
were of average difficulty. However, for test cases that posed a higher
complexity level, sophisticated measurement tools were still required.
Nevertheless, due to test cases mostly being of an average difficulty level,
they acknowledged that the new system would allow more efficient test
analysis.

"I understand that the new system is not made to replace our existing tools
but to add a layer of providing overview which is nice. I still do believe
that for in-depth detailed analysis, the old system has the power of the tools
to dig in, whereas the new system, is better for average cases which are
indeed more in number and the new system gives a quick overview to testers."
\- An Experienced Tester’s feedback on both systems.

The less experienced subjects expressed that the time-consuming tasks they had
to perform to begin the test analysis and diagnostics, like accumulating files
and information for a test object, were tedious and repetitive. The new
system, however, made such tasks simpler to perform. Hence they believed they
would benefit as their primary focus would remain on understanding and finding
issues rather than spending time preparing the diagnostic data for analysis.

"The testing tools we use have a learning threshold in the old system compared
to this new system which is more intuitive and user-friendly. In the old
system, it can be challenging to find the correct information because there
are a lot of software and other tools that sometimes is used for test analysis
and diagnostics. The new system on the other hand make it more appealing to
the less experienced person and gives them a chance to slowly dive into our
work."  \- An Experienced Tester’s feedback on both systems.

Lastly, the subjects agreed that the new system was simpler to understand.
Learning a complicated system was not a good start to their journey as a
tester. However, they suggested that, for difficult cases, they would use both
the new and the old system. They revealed that features such as structuring of
the files and its information could be quickly understood in the new system,
while analytic precision could be gained from the old system.

"All the information needed is accessible from the left side bar menu and it
is intuitive and easy to understand. Here, I don’t have to access three
different systems to get an overview information of the test object and test
case and these information when found faster makes it easier to do the job."
\- An Inexperienced Tester’s feedback on new system.

Table 5.9 summarizes all the findings from the usability test interviews. The
table explains the categories that were made using Nvivo, which were used to
collect findings in the interview. Each category has a description summarizing
the findings. The pictorial representation of the table can be seen in
Appendix J.

Table 5.9: Major findings from the decoded interviews after usability tests with short description. Findings from decoded Interviews |  Description
---|---
Files structured efficiently and stored in a single domain |  Testers explained that the new system created a framework that provided a much-needed structure to the files related to test object, test cases, and test diagnostics.
Structuring improved accessibility of information and files |  The testers addressed accessibility concerning time (files and information can be accessed faster) and collectively gathered them.
The old system preferred for in-depth analysis |  The testers explained that, for difficult test cases, they would need to use the old system, which granted them a more sophisticated set of tools.
The new system preferred for overview and average test cases |  For average test cases and getting an overview of the information in all cases, the new system was helpful.
Time for locating information reduced |  Information, in general, was more comfortable to locate in the new system due to the better structure.
Testers got the overview of the information faster |  The new system gives the testers a faster overview of the information they require to either start their evaluation of the diagnosis or come to a quick conclusion.
Mistakes due to interpretation or misinformation reduced |  Due to all the files and information being readily accessible in the new system, lack of information or possibilities of misinformation was reduced.
Information and data presented efficiently for analysis |  The presentation of the information and data were done efficiently, keeping it user-centered (Tester). Subjects could relate to the information presented in the user-interface.
Traceability of historical data improved |  Testers now had the possibility of tracing testing related objects, information, and data due to the structure the new system provided.
Files related to testing and diagnostics were accessible faster |  Testers need for requiring such files was always crucial, and the new system made this easier for them in terms of time.
The effort for navigation for data reduced |  Testers did not need to navigate between various tools to make sense of the test-data and various information.
New system easier to use for less experienced testers |  Due to the ease of use of the new system, a consensus emerged, describing it to be an attractive choice for testers new to the automotive testing field.

####  5.4.4 Task Measurement Results

The objective measure (i.e., time for task completion in minutes) for each
task performed in both systems are tabulated in Table 5.10. The table
indicates the time taken for the test subjects to complete each task
(explained in section 4.5.1). In the new system, the overall time taken to
complete all four tasks is significantly reduced compared to the old system.
The testers’ experience of the task, along with analysis using the NASA TLX
will be described in this section. The scores range is from 0-100, where
scores closer to zero suggest low task demands and scores closer to 100
suggest high task demands. Testers 1, 2 and 3 were inexperienced, whereas
testers 4, 5 and 6 were experienced.

Table 5.10: Time of completion for the Usability Test tasks in minutes. Tester | System | Task 1 | Task 2 | Task 3 | Task 4 | Total
---|---|---|---|---|---|---
Tester 1 | Old | 10 | 13 | 18 | 20 | 61
Tester 1 | New | 5 | 10 | 13 | 5 | 33
Tester 2 | Old | 5 | 10 | 20 | 10 | 45
Tester 2 | New | 3 | 7 | 17 | 2 | 29
Tester 3 | Old | 10 | 5 | 10 | 20 | 45
Tester 3 | New | 6 | 3 | 7 | 8 | 24
Tester 4 | Old | 11 | 10 | 15 | 15 | 51
Tester 4 | New | 5 | 10 | 15 | 8 | 38
Tester 5 | Old | 15 | 10 | 18 | 15 | 58
Tester 5 | New | 10 | 4 | 16 | 5 | 35
Tester 6 | Old | 12 | 11 | 20 | 17 | 66
Tester 6 | New | 6 | 4 | 10 | 13 | 33

The usability test was performed with four different tasks, where every task
was associated with requirements mapped in Table 5.11. Results of the
usability test show the most improvements for task one, two and four.

Table 5.11: Usability test task mapping to requirements. Task | Task Name | Requirement
---|---|---
Task 1 | Understanding the issue | UR1, UR8, UR9
Task 2 | Gathering information and data | UR2, UR6, UR10, UR11, UR12
Task 3 | Solving the issue | UR3, UR4, UR5, UR13, UR14, UR15
Task 4 | Reporting the issue | UR6, UR7

#####  5.4.4.1 Task 1 Results

The testers, combined, spent 63 minutes on Task 1 in the old system and only
35 minutes in the new system. All the testers expressed that the data
structuring had improved significantly and provided them with an overview of
the information they needed to proceed with the test diagnostics and analysis.
Time spent on Task 1 for each tester can be seen in Figure 5.6.

T1T2T3T4T5T6$0$$10$$20$$30$$10$$5$$10$$11$$15$$12$$5$$3$$6$$5$$10$$6$TestersTime
taken in minutesOld SystemNew System Figure 5.6: Time taken to complete Task 1
by the testers (T), in both systems.

"The new tool give me a very good overview of all the basic information I need
for a test case. This saves time for getting the overview of things making it
faster for me to make decisions."  \- Tester’s feedback on Task 1.

The scores from the NASA TLX for Task 1 shows that five out of six testers
felt an overall improvement on the workload. The testers’ NASA TLX scores for
Task 1 can be seen in Figure 5.7. Except for Tester 1 (T1), the workload score
was improved by more than 50% when using the new system. Tester 1’s score was
although not improved in the new system, but was somewhat worse. The tester
was more comfortable using the old system to perform Task 1 and did not see
much improvement with the new system. Tester 1 had less experience in test
diagnostics in general and was more used to the old system. It was also
noticed that the testers with less experience finished relatively faster than
the experienced tasters. The inexperienced testers did not reassess their
decisions while doing the different tasks in the test. In contrast, the
experienced testers were more cautious and repeated the tasks to evaluate
whether they made the correct choices.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$34$$53$$42$$56$$49$$45$$49$$19$$13$$25$$14$$15$TestersNASA
TLX scoreOld SystemNew System Figure 5.7: NASA TLX scores of Task 1 for each
tester (T), in both systems.

Mental Demand: The mental demand scores for Task 1 can be seen in Figure 5.8.
No matter what experience the tester had, the mental demand was lessened when
using the new system. By providing a better presentation and structure of the
data and information, the task was easier to perform, and the complexity was
reduced.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$120$$60$$40$$65$$65$$80$$55$$30$$20$$10$$30$$10$$15$TestersNASA
TLX mental demand scoreOld SystemNew System Figure 5.8: NASA TLX mental demand
scores of Task 1 for each tester (T), in both systems.

Temporal Demand: The temporal demand scores for Task 1 can be seen in Figure
5.9. In general, the testers found the new system to be more time-efficient
when understanding the test case context and getting an overview of an issue.
The ease of use made the task faster to complete. For Tester 1 (T1), this was
not the case because the tester was comfortable with the old system and
introducing the new system introduced the need to learn a new tool.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$25$$45$$50$$35$$40$$40$$75$$25$$10$$20$$25$$15$TestersNASA
TLX temporal demand scoreOld SystemNew System Figure 5.9: NASA TLX temporal
demand scores of Task 1 for each tester (T), in both systems.

Effort: The effort scores for Task 1 can be seen in Figure 5.10. The work
needed for Task 1 was greatly reduced in the new system. When the data is
presented in one tool, the navigation work was highly reduced. If there was
some crucial information the testers forgot and wanted to look up again, it
took less work for the tester to do in the new system. Once again, Tester 1
(T1) did not see any improvement in the effort score for the new system
because T1 was more comfortable with the old system.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$120$$30$$70$$35$$75$$50$$65$$40$$20$$20$$40$$15$$25$TestersNASA
TLX effort scoreOld SystemNew System Figure 5.10: NASA TLX effort scores of
Task 1 for each tester (T), in both systems.

Frustration: The frustration scores for Task 1 can be seen in Figure 5.11. The
stress and irritation level during the task was reduced greatly in the new
system. As the data was all in one tool, the tester was allowed to put all
focus on understanding the test case. The high effort score for Tester 1 (T1)
was due to general inexperience in test diagnostics and the requirement to
learn a new tool.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$20$$60$$25$$50$$20$$40$$60$$15$$15$$15$$10$$10$TestersNASA
TLX frustration scoreOld SystemNew System Figure 5.11: NASA TLX frustration
scores of Task 1 for each tester (T), in both systems.

In conclusion, having all the necessary and correct information in one place
reduced navigation work. When everything was categorized and presented in the
same tool, the effort needed was lessened. More effort could then be used to
understand the test case. The likelihood of finding incorrect information was
reduced in the new system as the most relevant information was presented
directly.

"I also want to add that instead of having a lot of different tools and
software windows up, but everything in one window where you can navigate
around to find the required information can be good in terms of correctness as
you can go back to the information easier and not get lost in all the other
windows."  \- Tester’s feedback on Task 1.

"Misinterpreting the requirements, using wrong databases will give faults,
sometimes when you don’t put the signals in the correct configuration, then
the solution may look okay, but it is wrong. Having the wrong information
regarding the database or configurations is a major source of errors. There
are ways to get things in the old system, but you need to know a lot of
details to find things. But in the new system, all this information is here
and easily accessible."  \- Tester’s feedback on Task 1.

#####  5.4.4.2 Task 2 Results

Task 2 involved collecting all the necessary files and data for the test
object. The time taken summed to 59 minutes for all six testers in the old
system. The testers’ revealed that the unnecessary time was spent searching
for files manually. The testers completed Task 2 in 38 minutes in the new
system, a decrease of 21 minutes, as can be seen in Figure 5.12.

T1T2T3T4T5T6$0$$10$$20$$30$$13$$10$$5$$10$$10$$11$$10$$7$$3$$10$$4$$4$TestersTime
taken in minutesOld SystemNew System Figure 5.12: Time taken to complete Task
2 by the testers (T), in both systems.

"It feels like the new system has done a lot of the groundwork that I would
have done manually in the old system. The data and information being in one
place and is easy to follow. It is definitely more of a smoother experience
when doing test analysis."  \- Tester’s feedback on Task 2.

The testers’ explained that the files related to the test object were simple
to locate and download into their systems from the categorized menus in the
new system (see Figure H.7). They also reflected on how they could find the
information needed to proceed with testing in the new system using the
Information and DVM menu (see Figure H.6 and Figure H.3), without having to
spend a lot of time. One of the most crucial reasons why retrieval of data was
an issue was because the partnering company practised the storing of test
output data in a decentralized way. One key aspect regarding decentralized
systems is that they require a decentralized meta-data management system which
would make the location of required data transparent to the users[41].
However, in the case of the partnering company, the coordination system was
not optimally implemented or used effectively, hence making it extremely
difficult for the stakeholders to access all the necessary data for a test
object in an efficient way. As the data was scattered across different
locations, it took more time and effort to collect all the required data that
represented a test object. Hence, showing the data in logical categories and
web pages in the new system place allowed data and information retrieval to
become faster.

Regarding the results of the NASA TLX scores for Task 2 in the old system (see
Figure 5.13) the overall score for the three inexperienced testers was 74, 74
and 55 respectively and 30, 38 and 46 respectively for the experience testers.
The less experienced testers seemed to struggle more than their experienced
counterparts. The interviews and observation uncovered the following findings.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$74$$74$$55$$30$$38$$46$$13$$13$$14$$22$$21$$18$TestersNASA
TLX scoreOld SystemNew System Figure 5.13: NASA TLX scores of Task 2 for each
tester (T), in both systems.

Mental Demand: The mental demand scores for Task 2 can be seen in Figure 5.14.
For the inexperienced testers, the mental demand was consistently high,
resulting in 75, 70 and 70 for the old system. For the experienced testers,
the scores were 20, 60 and 60 for the old system. Despite using the old system
for at least a year, five out of the six testers believed that the task was
high in mental demand. The testers collectively blamed the constant context
switching between different applications and storage systems to collect the
necessary data while also maintaining correctness in terms of data collection.
The scores were relatively lower in the new system. The inexperienced testers
scored 5, 10, and 15, while their counterparts scored 20, 25, and 20 for the
new system. The testers explained that the menu categories in the new system
made it simpler to interact with and gain the needed information without
context switching between different applications. Here, it can be seen that
the mental demand score for the new system was low relative to the old system.
However, the scores were still slightly higher in the new system for the
experienced testers compared with inexperienced testers. The experienced
testers revealed that as they had a habit of making sure they were working
with the correct information, and they reevaluated their choices often. The
inexperienced testers, however, relied on the system to be correct with the
data and did not reassess their choices.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$75$$70$$70$$20$$60$$60$$5$$10$$15$$20$$25$$20$TestersNASA
TLX mental demand scoreOld SystemNew System Figure 5.14: NASA TLX metal demand
scores of Task 2 for each tester (T), in both systems.

Temporal Demand: The temporal demand scores for Task 2 can be seen in Figure
5.15. The inexperienced testers found it overwhelming to perform this task in
the old system as it required constantly moving between various applications
and looking for the required data. On top of that, due to their low experience
level, they had to frequently confirm whether the outcome of the task resulted
in desired data, hence making it more time-consuming. For the experienced
testers, the task appeared to be slightly trivial, as they were used to it in
their day to day work. However, they acknowledged struggling with such type of
task during their early stages of being a tester. The inexperienced testers
had a significantly lower score with the new system, accounting for a score of
15 for each. They explained the lack of context switching between testing
tools provided an immense advantage because all the necessary data were
categorized accordingly in the new system. As the data was categorized,
presented, and labelled in the new system’s interfaces, navigation and
searching for information felt easier. The testers’ explained they no longer
had to apply guesswork or rely on intuition to collect data for the test cases
as they were well structured and traceable in the new system. Hence, this
reduced performance time, giving them more time to conduct tests analysis. As
for the experienced testers, all of them scored 20 each in the new system,
stating the same reasoning as the Inexperienced testers.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$85$$60$$50$$20$$20$$35$$15$$15$$15$$20$$20$$15$TestersNASA
TLX temporal demand scoreOld SystemNew System Figure 5.15: NASA TLX temporal
demand scores of Task 2 for each tester (T), in both systems.

Effort: The effort scores for Task 2 can be seen in Figure 5.16. The
inexperienced testers scored 70, 80 and 50 respectively in the old system.
They blamed the manual process of the task. The testers all agreed that the
old system was not tailored for the inexperienced testers in terms of
usability. The consensus amongst the testers suggested that they had to spend
additional efforts adapting to the flaws of the old system, along with the
task, hence increasing the time and labour. As for the experienced testers,
they had slightly better scores ranging from 30 to 50 in the old system.
Despite their knowledge and practice with the existing system, they still
believed that it was not a pleasant experience. Intuition played an important
role when they performed the task because they could anticipate where and what
to find. Despite that, they found certain aspects of the task like the data
and file collection for the test cases a nuisance rather than a challenge. For
the inexperienced testers, the scores were much better in the new system. They
explained that certain aspects of the new system made the overall task and its
experience intuitive and simple. Amongst those were the neatly categorized
menus which held all the necessary information and data for the test case
logically and concisely. Hence, the inexperienced testers felt that the
interface brought a sense of predictability in regards to what they wanted to
see, making the task consume less effort. As for the experienced testers, they
too scored better here, accounting for 35, 15 and 20 in the new system. They
discussed the implementation of the interface and how it saved them time on
collecting data, files, and information which should have always been at the
testers’ disposal.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$70$$80$$50$$50$$30$$45$$10$$15$$15$$35$$15$$20$TestersNASA
TLX effort scoreOld SystemNew System Figure 5.16: NASA TLX effort scores of
Task 2 for each tester (T), in both systems.

"In the current way of doing this (old system), when there is new project or
unfamiliar test cases, I have to pinpoint a lot of the things such as version
numbers, databases etc myself, so it can be kinda tedious as I have to go
around and find it on documents and web pages. The new system provides a lot
of this stuff, so I think this can be very valuable ."  \- Tester’s feedback
on Task 2.

Frustration: The frustration scores for Task 2 can be seen in Figure 5.17. The
inexperienced testers altogether found this task extremely frustrating with
the old system. Their TLX scores were 70, 85 and 65 individually, suggesting a
rather frustrating experience overall due to the redundant routine of manually
accumulating test-related data. The experienced testers scored less,
accounting to 30, 30 and 65 individually in the old system. As they had more
experience of gathering such data for test cases, they could intuitively
realize whether the data they collected was correct or not. Both groups of
testers explained that they almost always had to go through this routine each
time they faced a new test case or during debugging/re-running test cases
which were performed a long time in the past. Hence, finding the correct data
was always a challenge because data could be missing or not archived for more
than a certain period, making the whole process very frustrating.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$120$$70$$85$$65$$30$$30$$65$$20$$15$$10$$15$$20$$15$TestersNASA
TLX frustration scoreOld SystemNew System Figure 5.17: NASA TLX frustration
scores of Task 2 for each tester (T), in both systems.

The TLX scores for the inexperienced testers were better in the new system.
They scored 20, 15 and 10 for frustration. The experienced testers group had
relatively less deviation, apart from one who scored 15 (65 in the old
system). They claimed to have exerted significantly less effort due to the
information and data being appropriately structured. Even though the proposed
system was new for them, it proved to be intuitive in terms of navigation.
They said that the interface was mostly self-explanatory, which made the task
not frustrating. The need for data to be in a common platform was crucial from
their perspective. Locating the necessary data to make decisions on a test
case would always require time and unnecessary work.

"The experience was smooth and quick overall. Not much time needed to look for
stuff. You know what I mean? We as testers need to get some information on a
fast basis and the tool gives that overview quickly."  \- Tester’s feedback on
Task 2.

#####  5.4.4.3 Task 3 Results

Task 3 involved readying the testing tools using the data/files from Task 2
and creating an overview of information about the test to find the root cause
of the failure. The total time spent on Task 3 (see Figure 5.18) on the old
system summed to 101 minutes and 78 minutes for the new.

T1T2T3T4T5T6$0$$10$$20$$30$$18$$20$$10$$15$$18$$20$$13$$17$$7$$15$$16$$10$TestersTime
taken in minutesOld SystemNew System Figure 5.18: Time taken to complete Task
3 by the testers (T), in both systems.

According to the testers, the typical issues while performing this task
included the time and effort spent in preparing the tools for analysis,
navigating between the multiple tools and files and lastly, working with large
and unstructured data for test cases. In the new system, the efforts needed to
prepare tools was no longer an issue as testers(mostly Inexperienced) agreed
that the files and data required to prepare the tools were present in the new
system on the dashboard. All of them agreed that the time for locating
information was also reduced as they did not have to switch between tools to
find what they needed.

"Faster overview(in the new system), helps give insights to the problem, and
saves time from the eventual analysis in the analysis tools."  \- Tester’s
feedback on getting a good overview.

Furthermore, the information was presented efficiently and intuitively based
on the requirements of the testers, giving them more time to focus on
analyzing the test case rather than doing unnecessary work. The testers
finally concluded that finding issues depended on the experience and the two
systems did little to help there apart from aiding to set up the analysis.

"You can make a decision on easier test cases I think directly in the new
system. When you see the expected signal values in the new interface, you can
see from the pattern directly in the graph and find what went wrong which
could save time. For more complicated analysis, this can also help you get a
head start but for in-depth analysis the old system provides more tools."  \-
Tester’s feedback on finding issues in tests.

The NASA TLX scores for Task 3 can be seen in Figure 5.19. The NASA TLX scores
in the old system for the experienced testers were 66, 55 and 34 and 69, 18
and 51 for the inexperienced testers. However, in the new system, the
experienced testers dropped to 52, 37 and 24 while the inexperienced fell to
55, 18 and 39, respectively. The new system did reduce the workload compared
to the old system, but only slightly. This is because the actual sensor signal
analysis in the current tools offers a lot of advanced features, and if the
testers know how to utilize them, they made it easier to complete the task.
For simpler tasks where those advanced features were not needed, the testers
could use the new system for faster judgment and decisions.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$66$$55$$34$$69$$18$$51$$52$$37$$24$$55$$18$$39$TestersNASA
TLX scoreOld SystemNew System Figure 5.19: NASA TLX scores of Task 3 for each
tester (T), in both systems.

Mental Demand: The mental demand scores for Task 3 can be seen in Figure 5.20.
As the task required to finding issues in test cases, their ability to solve
this task was not the major drawback for the testers. It was rather the
inconvenience of needing to find things they needed in various systems.
Perceptual activities like constantly needing to move between windows of
different tools made the task cumbersome and mentally taxing. In the old
system, the inexperienced testers gave a score 85, 60 and 70 while the
experienced testers gave 80, 20 and 55, respectively. The testers at the
initial stages of the task did the most context switching between tools using
the old system, which was cumbersome, mentally frustrating and annoying. In
the new system, however, the testers spent less time gathering an overview of
the task due to the information being present already in the overview section.
The new system did not explicitly help them in identifying the root-cause as
this depends on the experience and knowledge of the tester. Still, it reduced
the mental effort by decreasing perceptual activities and context switching.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$120$$85$$60$$40$$80$$20$$55$$65$$30$$30$$70$$20$$40$TestersNASA
TLX mental demand scoreOld SystemNew System Figure 5.20: NASA TLX mental
demand scores of Task 3 for each tester (T), in both systems.

"The new tool gives me a very good overview of all the basic information I
need for a test case. This saves time for getting the overview of things
making it faster for me to make decisions."  \- Tester’s feedback for Task 3.

Temporal Demand: The temporal demand scores for Task 3 can be seen in Figure
5.21. The testers noticed that, if they could not create an overall
understanding of the task rather quickly, it would hurt their chances of
finding the root-cause. The old system had an overall higher score for
temporal demand due to that. As the task was time-boxed, the testers felt a
sense of urgency while accomplishing the task in the old system. In the new
system, however, the time required to develop and comprehend an overview of
the artifacts was less. The testers also felt fewer time constraints in the
new system. The experienced testers noted a reduction of time needed to locate
information and gather an overview, leaving more time for root-cause analysis.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$120$$75$$65$$50$$75$$25$$70$$65$$30$$35$$35$$25$$30$TestersNASA
TLX temporal demand scoreOld SystemNew System Figure 5.21: NASA TLX temporal
demand scores of Task 3 for each tester (T), in both systems.

Effort: The effort scores for Task 3 can be seen in Figure 5.22. For the
effort, the inexperienced testers scored 75, 65 and 25 in the old system. The
first two testers suggested that the effort to find the required information
from within the different tools presented itself as a hurdle in the task. For
the experienced testers, the numbers were similarly high (70, 25 and 75) in
the old system with the same feedback as the inexperienced testers. Another
reason for such high effort was due to the time spent in not misinterpreting
the information they found from the different tools and files. All the testers
admitted that the new system was good for getting an overview and a better
choice in attempting to solve easy-medium difficulty test cases, however, they
would still prefer the old system for in-depth analysis. Even though the new
system gave them easier access to information for the test artifacts, finding
the root cause of the issue was not made easier with this change. The new
system helped the testers to locate and utilize data quickly and efficiently
but did not improve the testers ability to think. The mental efforts needed to
find the root-cause would always remain high as that is based on testers
experience.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$120$$75$$65$$25$$75$$20$$50$$70$$50$$25$$75$$15$$50$TestersNASA
TLX effort scoreOld SystemNew System Figure 5.22: NASA TLX effort scores of
Task 3 for each tester (T), in both systems.

Frustration: The frustration scores for Task 3 can be seen in Figure 5.23. The
most frustrating aspects of the task for the testers were finding the needed
data, artifacts and their information from the files, before attempting to
find the root cause in the old system. Everyone agreed that they were
habituated to the old system to the point that the once frustrating aspects
that caused them trouble during the starting of their career eased eventually.
All three inexperienced testers thought that the new system reduced efforts
for navigation of data, reducing time for locating information. One tester
opted to give a slightly higher value (from 35 to 40) because the new system
was something that the person needed to get used to. For the inexperienced
testers, the score was practically the same. The new system did not help them
deal with the frustration encountered when attempting to identify the issue in
the test case. A common consensus here was that the required knowledge for
finding root-causes was still unique to the individual testers, which is
something that the new system does not improve. They did agree that it aided
them during the task.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$120$$60$$35$$30$$60$$15$$45$$30$$40$$15$$50$$15$$45$TestersNASA
TLX frustration scoreOld SystemNew System Figure 5.23: NASA TLX frustration
scores of Task 3 for each tester (T), in both systems.

#####  5.4.4.4 Task 4 Results

Time spent on Task 4 was greatly decreased in the new system compared to the
old system. Introducing a feature where where all the stakeholders involved or
responsible are listed in the new system was appreciated by the testers as
they now had an option to narrow down which stakeholder they should contact to
perform this task. The time taken for each testers for Task 4 can be seen in
Figure 5.24. The total time for the testers in the old system was 97 minutes
and 41 minutes in the new system, making it more than 50% more efficient.

T1T2T3T4T5T6$0$$10$$20$$30$$20$$10$$20$$15$$15$$17$$5$$2$$8$$8$$5$$13$TestersTime
taken in minutesOld SystemNew System Figure 5.24: Time taken to complete Task
4 by the testers (T), in both systems.

"The stakeholder view did stick out (in the New System) and I think it could
be very useful as it is lacking in the old system."  \- Tester’s feedback on
Task 4.

The NASA TLX scores in Task 4 can be seen in Figure 5.25. The scores were
greatly improved in the new system compared to the old system for all the
testers. In the old system, the information about stakeholders was not easy to
find because it required backtracking to retrieve the information, and the
experience and knowledge of the tester were needed to know where to find the
information about stakeholders within the company or department. By presenting
a list of stakeholders in the new system, the work effort was significantly
reduced for the testers as the work was done by the tool itself. From the
stakeholder list, the testers could make decisions on who was the most
suitable stakeholder to contact regarding the test case. Even if an incorrect
stakeholder was chosen, it was straightforward to choose another more suitable
stakeholder from the stakeholder list in the new system.

T1T2T3T4T5T6$0$$20$$40$$60$$80$$100$$35$$54$$49$$68$$38$$51$$19$$33$$20$$23$$22$$18$TestersNASA
TLX scoreOld SystemNew System Figure 5.25: NASA TLX scores of Task 4 for each
tester (T), in both systems.

The stakeholder list provided a good additional feature to support testers
decision making. By narrowing down and eliminating stakeholders until only the
most suitable stakeholders were left, testers could complete this task without
much work effort.

"In the old system it’s actually hard to really know who is responsible and I
usually just go to my product owner and ask him who to contact. By providing a
list of the stakeholders, it makes it easier to narrow it down."  \- Tester’s
feedback on Task 4.

#####  5.4.4.5 Summarizing Task Improvement Results

Tasks 1, 2 and 4 showed more improvement in terms of task load index and time
consumption scores than task 3. The reason was that task 1, 2 and 4 consisted
of a lot of repetitive manual tasks which were difficult to perform in the old
system. Task 3 required the testers to investigate root-cause issues in the
test case, which depended a lot on how the testers evaluated the test results
and situation. Aspects of Task 3 related to inspecting information were
reduced in effort and time consumption. However, the thought process of each
tester was unique and did not change when using either system. As a result,
there was less improvement for Task 3 from the use of the new system.

## 6 Discussion

This chapter starts with a discussion on the research questions of this study
based on the results from Chapter 5. A framework will also be presented to
improve automotive testing in the context of data presentation. Lastly, future
work to extend this study will be discussed along with threats to the validity
of this study.

###  6.1 Deteriorating Factors for Decision Making

Recall RQ1: What aspects deteriorate decision making during test analysis and
diagnostics of automotive electronics?

Various aspects lead to deterioration of decision making which were gradually
uncovered throughout this study. Looking at the current challenges in test
analysis and diagnostics for automotive electronics at the partner company,
unstructured data and superfluous information were the primary challenges.
Working with multiple tools, storage, and systems made finding and using the
necessary information to conduct test analysis difficult. The causes of such
problems were also related to excessive navigation between different tools and
working with an abundance of information. All the findings that helped answer
the RQ1 were categorized and can be seen in Table 5.3. Factors that made
decision making more complicated than necessary for the testers were
thoroughly investigated in the problem identification phase through
interviews, questionnaires, observations and in the evaluation phases. The
deteriorating factors were as follow:

  * •

Spending unnecessary time doing redundant things(such as manually collecting
related files) in the testing process.

  * •

Lacking a good presentation of test data to create an overview for the
testers.

  * •

Not storing test-related data in a standard way.

  * •

Needing to traverse and navigate large unstructured data and information using
the different tools all at the same time.

  * •

Needing to locate stakeholders to aid with fault mitigation of test cases.

The testers workload and focus were used primarily to retrieve the relevant
data and information to conduct test analysis and diagnostics. The correctness
of the data was affected when it was not easily accessible. If the testers
wanted to double-check the information to confirm its correctness, their
workflow was slowed because of the difficulty in accessing data. By providing
accessibility to the data, the workload and focus were instead shifted to the
actual test analysis and diagnostic, making it easier for the tester to put in
more effort on during analysis.

###  6.2 Design of Test Analysis Framework with Decision Making Models

Recall RQ2: How should the design of a test analysis framework for automotive
testing reflect the decision making models followed by testers?

The testers mostly had a typical process of thinking and making decisions in
the old test system. For completion of each test task, the testers relied on
the related data to decide the outcome. However, as data was not efficiently
available to the testers, they could not make informed decisions effectively.
For example, to begin the test analysis process, testers had to read and
understand the requirements and collect all the files for starting the test
analysis. The files were not easy to find because the testers had to search
for them manually from an abundance of other files and eliminate the
unnecessary ones. The approach and design of this process were not efficient
as it increased effort to complete the tasks. To improve the design of the new
system, we opted to understand and map how the testers perform the
requirements during the test analysis process and what kind of decision making
model best described their decision approach. Data for requirements that were
mapped with the elimination by aspects model were placed in the new system
where the testers would find it with the least amount of searching and
elimination of unnecessary information possible.

The data for the requirements that were mapped with the additive/linear model
and the conjunctive model were logically categorized in various views(web
pages) in the new system. Each view represented a specific type of data. For
example, the data related to requirements, test case steps, test output files
(log, signal) and stakeholders were separated into separate web pages. Through
evaluations of the design, the testers agreed that the design choice allowed
them to better understand the data, which aided decision making because the
information structure and presentation did not overwhelm or confuse them. By
logically categorizing the data, the accessibility of information and decision
making satisfaction increased.

Due to the logical categorization of information into their separate views,
the navigation of information improved in the new system. The decision making
models helped us to design the features of the requirement to the new system
in an efficient manner to aid decision making by understanding the tester’s
decision making process for each feature and design the data structure
accordingly.

###  6.3 Structuring and Presenting Data

Recall RQ3: How should data be structured, presented, and documented in this
test analysis framework to improve the quality of information available during
test analysis?

Through evaluation and interviews with the testers, the issues that affected
the old system regarding structuring test data and files, presenting test-
related information and data efficiently and documenting stakeholders were all
identified. Improvements were implemented by building prototypes of a new
system. The low fidelity prototypes were improved upon by evaluating them with
expert testers, finally coming to a consensus on the user interfaces,
experience and data presentation. To ensure a better structure for data, we
opted to take inspiration from web-based dashboards because they are intuitive
in design and easy to understand. In the dashboard, we chose to present
similar types of data as logical groups.

As testers needed to use a lot of test-related data, the data was neatly
organized in the left sidebar as can be seen in Figure G.1. Each category in
the sidebar revealed a new interface with additional information and data. The
requirements for the test cases can be seen in Figure G.2 and the signals from
the test object for test data analysis and finding the root cause can be seen
in Figure G.5 and Figure G.6. The prototype made using Figma was later was
converted to React.js to improve usability, and it became the final high
fidelity prototype which was later used in the usability tests. The reason for
opting to create the final high fidelity prototype using React.js was to make
sure that, during the usability tests, the testers felt as if they were
interacting with a web-based application. Throughout the development of the
prototype, the discussions we had during the evaluation phases helped identify
improvements. Efficiently structuring data and presenting it had a positive
impact on the testers ability to solve issues in test cases. They acknowledged
that the implemented improvements helped aid them in analyzing test cases by
providing testers with a good overview of information. Table 5.9 describes the
findings from the usability tests.

The requirements that were developed and the design of the new system can be
used as an inspiration to structure and efficiently present data for test
analysis. This study showed that the new system was more time-efficient and
required less effort for the testers to perform testing related tasks. General
guidelines on how to structure and present data to support decision making and
communicate verdicts efficient for test analysis of automotive electronics are
as follows:

  * •

Use a familiar design for the tool used by the testers such as a web-based
system or dashboard.

  * •

Structure and present data of different categories in different views to not
overwhelm the tester. For example, the test requirement should be in one view,
while test object data should be in another view.

  * •

Give an overview of the test case as the landing page to help testers
understand the context more efficient. Include the result of the overall test
case and each step, test case dates and other essential data about the test
case.

  * •

Offer relevant data that can be used for test analysis easily accessible, such
as functional and test requirements, test object information and measurement
configuration.

  * •

Include revisions numbers on the data and information. Revision numbers are
the different versions or states of the objects, data or information. If a
requirement gets updated for instance, a new revision number is assigned which
is usually an increment of the previous revision number.

  * •

Include a list of stakeholders with contact information that are involved or
relevant for the test object, requirements and functionality.

  * •

Show the different test sequences or steps in the test case with an overview
of the output. The output can be a summarized graph with sensor signal values
and information.

###  6.4 Accessibility of Test Analysis

Recall RQ4: How can this test analysis framework be designed to support ease
of use and accessibility of test related data during test analysis?

The data used for test analysis should be placed in one tool that is easy for
the testers to access. Having one tool instead of multiple tools increased the
navigation efficiency between pieces of analysis data. By giving the testers
the data related to requirements, test object, test cases, test diagnostics
and stakeholders in one single tool, we increased the overall accessibility
and efficiency of data collection. The categorization of data and presenting
them into separate views in the same tool according to its association also
increased accessibility because the testers could easily find and access the
data they needed for the test analysis.

Because we used a common web-based design for the system, the testers felt
more comfortable with using the new tool. The motivation of using a web-based
design was because it was commonly used for decision making systems [22] and
is well known for all internet users. The design was intended to be user-
centered and focus on the tester’s needs for test analysis and diagnostics.
The testers’ responses from the evaluation phases and usability test indicated
that the new system’s design was easy to work with and made it more efficient
to access the data needed for test analysis.

An interesting finding in this study was how much the testers appreciated the
new system with its improved navigation. By providing more efficient
navigation while working with large unstructured data, it made it easier for
the tester to confirm their findings before making a decision, which reduced
potential misinterpretation of information. In the study performed by Bharati
and Chaudhury [21], information presentation does not have a positive effect
on decision making satisfaction where navigation efficiency is a part of
information presentation. Their research was conducted for customers decision
making during shopping. Given the different context between their work and
this study, navigation efficiency was a useful factor when working with a
large amount of unstructured data as it can sometimes be overwhelming for the
user to remember everything. Information presentation also includes graphics,
color and presentation style, but these factors were not something that was
noticed by the testers in this study.

###  6.5 Summarizing the Findings

The findings revealed various problematic aspects in both the old and the new
system. All testers agreed that information accessibility increased due to
better structuring of the test-case related files and data. As a result, the
time required to locate data reduced significantly. This can be seen in Figure
5.12 that depicts the time taken for the Task 2 specifically. The presentation
of information and data increased the perceived value of the new system.
Improved presentation of data resulted in a decrease in time and effort for
Tasks 1, 2 and 4. Information was accessible faster with little to no context
switching, which helped in decreasing misinterpretation of information. All of
this was achieved due to fulfilling user requirements 1-10. The benefits of
the new system can be summarized in Figure 6.1.

Figure 6.1: Summarizing for the findings from the interview after usability
test.

The testers agreed that the new system would be a better choice to analyze
test cases with easy-to-medium difficulty. The new system would also be a
better option for testers who were reasonably new to testing. It would be a
stepping stone for them, which would gradually train them to move into more
sophisticated tools and systems eventually.

"It’s a higher learning threshold in the old system compared to this new
system which is more intuitive and user-friendly. In the old system, it can be
challenging to find the correct information because there is a lot of software
and other tools that sometimes is not that useful for test analysis and
diagnostics. The new system on the other hand make it more appealing to the
less experienced person and gives them a chance to slowly dive into our work."
\- Tester’s feedback on the new system.

The old system, however, had the advantage when dealing with difficulty test
cases. The testers, regardless of their experience, considered the old system
to be better at a low-level analysis of test cases. The need for further
analysis requires working with tools that offer precision. If they needed to
work their way around a difficult test-case, they could use the new system to
retrieve and visualize data and use the old system for analyzing issues and
finding root-causes. Analyzing test cases also requires experience and
knowledge on the domain, which neither of the systems can provide as they are
merely tools at a tester’s disposal. However, the new system aided the overall
process of testing. The testers felt that the four individual tasks felt more
relaxed compared to the old system because they did not have to perform
several repetitive and trivial tasks in the new system.

###  6.6 Benefits and Implication On Practice

Figure 6.2: The framework based on design science methodologies used in the
study.

Many problems and challenges could be investigated in the domain of testing in
the automotive industry. Little work has been done regarding how individuals
perform tasks and how specific task workload affects their work in the
automotive industry. In this study, an approach to improve task performance
and the workload was conducted by improving the data structure and
presentation. The methodologies used in this study to find improvements can be
presented in a generic framework (see Figure 6.2) that companies can use to
identify areas to improve in regards of data structure and presentation.
Companies can use this framework as a guide to investigate problems and
challenges within the testing domain in their existing business process by
carrying out internal research. Due to the abstract nature of the framework,
we believe that it can be adapted and used in similar domains to improve and
overcome similar challenges investigated by this study.

The requirements we developed along with the testers can also help create
awareness of the problems testers face and how those problems can be rectified
by implementing solutions that help elevate the stakeholders. The findings
from this study also show areas where performance in automotive testing can be
improved by creating awareness of unstructured data, inefficient test analysis
practices, presentation of test data and advantages/disadvantages of the
testing tool within the testing domain. By starting on one area to improve the
unstructured data, such as testing analysis, the awareness can be spread to
other areas with challenges of unstructured data within the company. General
guidelines based on the findings of this study to improve test analysis
challenges are as follows:

  * •

Introduce a standard method for stakeholders to store and manage data. When
there is no standard way to store data, it will require more time and effort
to find the relevant data.

  * •

Use the right tools for the task and minimize the usage of unnecessary tools
to decrease navigation and context switching during test analysis as this can
be overwhelming for testers and can lead to loss of concentration.

  * •

Communication between stakeholders can be improved by mapping responsible
stakeholders to their relevant object, artifact or domain to ask for aid
efficiently.

  * •

Requirements revisions can be tracked to reduce misinterpretation of
information during test analysis.

The unstructured nature of the data is not an exclusive challenge in the
testing domain as testing analysis. This study looks into how the data
structure and presentation can be used as a solution to improve task
performance and workload when interacting with a system. The design and
evaluation methods used in this study can be applied to data visualization
research to identify further challenges regarding the relationship between
data presentation and workload performance on analytical tasks.

A previous study by H. Wang et al. [20] showed how better visibility of the
results and outputs of tests benefits testers with a quicker general
perspective. With this study, the same results were achieved by providing a
better overview of the results, outputs and information of the test case. We
also used NASA TLX as a tool to evaluate the performance to justify the
findings. We believe that the NASA TLX is a powerful tool to assess the
performance on users when conducting research for visualization tools to
improve effectiveness and efficiency of test analysis as it is essential to
understand the interaction of testers with the visualization tool as well.

###  6.7 Threats to Validity

####  6.7.1 Internal Validity

Controlling external factors such as participants fatigue levels when
performing usability tests is important, and this was done by giving them a
fixed recovery time between tasks. Still, different participants may need
different amounts of time to recover, and that could influence the performance
of the participants in the usability tests. We assumed that 15 minutes of rest
time would be sufficient for all participants. Another possible threat to
internal validity was that the participants in the usability tests, might have
had discussions with the testers involved in evaluating the prototypes. There
was no way for us to completely isolate the testers used for a usability test
and the ones used for evaluating the prototypes. However, we explained the
importance of such isolation to the testers beforehand. We also made sure that
learning effects were kept at a minimum by not having the testers repeat the
same specific task in both systems during the usability tests.

Qualitative coding of the results may be biased by our existing hypotheses
regarding the research questions, and gathered qualitative data could be
interpreted subjectively. We tried to minimize this effect by doing the coding
together without being biased towards our personal beliefs and perspectives.
We had discussions with each other so that the findings from the coding
sessions reflected the data with limited personal interpretation.

####  6.7.2 External Validity

The sample size for the interview, questionnaire, evaluation, and usability
test subjects could have been larger. Due to time constraints of the study, we
could not perform the usability tests with testers from other departments in
the company. We focused on one department (Power and Energy dept), which could
hamper external validity. To partially deal with this problem, we tried to
broaden our data collection to incorporate the knowledge from testers working
in various departments so that the new system would be created from a broader
perspective. Since this study was conducted with one specific industrial
partner, the generalization might be limited due to the particular setting of
the company. Due to time constraints, empirical evidence from usability tests,
the study of the existing testing process for testing and its impacts on the
stakeholders could not always be studied to a greater extent. However, to
mitigate this, we had multiple interviews with the same participants on
different days to evaluate consistency in their answers while taking the extra
time and effort to validate our understanding of their replies through follow-
up discussion. We believe the results of this analysis are applicable, at
minimum, at other companies in the automotive domain.

####  6.7.3 Construct Validity

As our research relies on using subjective measurements to answer the research
questions, there could have been times when we might have unwillingly and
unknowingly influenced the stakeholder actions. During interviews, we tried
not to affect the stakeholders’ answers, but we might have designed the
questions in ways that would help influence the resulting responses. Hence,
this could be a threat to construct validity because research biases may have
been reflected in the questions in the interviews. During the participant
observation, we tried to isolate ourselves from the participants completely.
Despite our best efforts, the feeling in the participants was noticeable
regarding, being watched and could have affected their way of working during
the observation session. Finally, there could have been additional test cases
for usability testing for both the systems because the participants could have
had better/worse scores in the TLX their feelings when analyzing a particular
test. The participants were given time to rest to minimize such effects.

###  6.8 Future Work

Cognitive research, especially regarding workload on individuals in the
testing domain within the automotive industry, has not been given the focus it
needs. Hence, further investigation in cognitive workload within the
automotive testing domain could be done to broaden the problems, and aspects
of the testing process addressed. The current research for testing of
automotive electronics and ECU testing has focused largely on generating and
optimizing the output data. With this study, we also hope there will be more
interest in examining other aspects, such as data presentation and structure.

Data structuring and presentation are aspects which can be further
investigated as this study showed improvements in terms of NASA TLX. The focus
should also be put upon how the data (files, sensor signal data,
documentation) is being used in an organization rather than within different
domains. Furthermore, the study can inspire researchers to take into
consideration the aspects of mental workload and how it affects decision
making.

## 7 Conclusion

In this study, we explored the testing domain in the automotive industry,
mainly focusing on testing of automotive electronics and its challenges in
regards to data structure and presentation. An exploratory case study was
performed with Volvo Cars, where we conducted multiple elicitation and
evaluation sessions to investigate the current challenges in data structure
and presentation within the testing of automotive electronics. Thirteen
stakeholders and six testers in the partner company were involved in this
study.

A prototype was created for this study to illustrate better data structure and
presentation. The design of the prototype was based on good design practices
and stakeholders’ feedback, knowledge, experience and evaluation results. The
final evaluation was conducted in the form of a usability test on the
prototype to gather data for analysis to answer the study’s research question.
The prototype’s results show that providing better data structure
significantly increases the efficiency and reduces the workload for testers
when conducting test analysis and diagnostics. The correctness of the data and
information was also improved by increasing the accessibility of the
information and aided with decision making during the test analysis.

In conclusion, managing data is a challenging task when there is an
overwhelming amount of data present. The management of overwhelming data will
always be a challenge for a company, and it can be not easy to have perfect
data management, structure and presentation. Still, challenges can be
minimized with the right practices. Good data structure and presentation is
essential to increase the efficiency and correctness of the testing process
for the stakeholders involved.

## References

  * [1] E. Bringmann and A. Krämer, “Model-based testing of automotive systems,” in 2008 1st international conference on software testing, verification, and validation, pp. 485–493, IEEE, 2008.
  * [2] K. Grimm, “Software technology in an automotive company-major challenges,” in 25th International Conference on Software Engineering, 2003. Proceedings., pp. 498–503, IEEE, 2003.
  * [3] Y. Onuma, Y. Terashima, and R. Kiyohara, “Ecu software updating in future vehicle networks,” in 2017 31st International Conference on Advanced Information Networking and Applications Workshops (WAINA), pp. 35–40, IEEE, 2017\.
  * [4] M. Jørgensen and S. Grimstad, “Avoiding irrelevant and misleading information when estimating development effort,” IEEE software, vol. 25, no. 3, pp. 78–83, 2008.
  * [5] R. Rao, “From unstructured data to actionable intelligence,” IT professional, vol. 5, no. 6, pp. 29–35, 2003.
  * [6] J. Colleaux and O. Sahraoui, “Bridging the gap between virtual and physical testing in electric & electronic automotive systems,” tech. rep., SAE Technical Paper, 2009.
  * [7] R. Blumberg and S. Atre, “The problem with unstructured data,” Dm Review, vol. 13, no. 42-49, p. 62, 2003.
  * [8] C. Ebert and C. Jones, “Embedded software: Facts, figures, and future,” Computer, vol. 42, no. 4, pp. 42–52, 2009.
  * [9] K. Lamberg, M. Beine, M. Eschmann, R. Otterbach, M. Conrad, and I. Fey, “Model-based testing of embedded automotive software using mtest,” tech. rep., SAE Technical Paper, 2004.
  * [10] M. Conrad, S. Sadeghipour, and H.-W. Wiesbrock, “Automatic evaluation of ecu software tests,” SAE transactions, pp. 583–592, 2005.
  * [11] M. Johanson, S. Belenki, J. Jalminger, M. Fant, and M. Gjertz, “Big automotive data: Leveraging large volumes of data for knowledge-driven product development,” in 2014 IEEE International Conference on Big Data (Big Data), pp. 736–741, IEEE, 2014.
  * [12] J. W. Payne, “Task complexity and contingent processing in decision making: An information search and protocol analysis,” Organizational behavior and human performance, vol. 16, no. 2, pp. 366–387, 1976.
  * [13] E. G. Leaphart, S. E. Muldoon, and J. N. Irlbeck, “Application of robust engineering methods to improve ecu software testing,” tech. rep., SAE Technical Paper, 2006.
  * [14] T. Foran and B. Jackman, “An intelligent diagnostic system for distributed, multi-ecu automotive control systems,” SAE transactions, pp. 395–402, 2005\.
  * [15] J. Novak and P. Kocourek, “Automated testing of electronic control units compatibility in vehicle can networks,” in ISIE Proceedings of the IEEE International Symposium on Industrial Electronics, vol. 4, pp. 1423–1428, 2005\.
  * [16] J. R. Hayes, “Human data processing limits in decision making,” tech. rep., ELECTRONIC SYSTEMS DIV HANSCOM AFB MA, 1962.
  * [17] J. Jacoby, D. E. Speller, and C. A. Kohn, “Brand choice behavior as a function of information load,” Journal of marketing research, vol. 11, no. 1, pp. 63–69, 1974.
  * [18] C. Hendrick, J. Mills, and C. A. Kiesler, “Decision time as a function of the number and complexity of equally attractive alternatives.,” Journal of Personality and Social Psychology, vol. 8, no. 3p1, p. 313, 1968.
  * [19] S. Lichtenstein and P. Slovic, “Response-induced reversals of preference in gambling: An extended replication in las vegas.,” Journal of Experimental Psychology, vol. 101, no. 1, p. 16, 1973.
  * [20] H. Wang, X. Zhang, and M. Zhou, “Mavis: Feature-based defects visualization in software testing,” in 2012 Spring Congress on Engineering and Technology, pp. 1–4, IEEE, 2012.
  * [21] P. Bharati and A. Chaudhury, “An empirical investigation of decision-making satisfaction in web-based decision support systems,” Decision support systems, vol. 37, no. 2, pp. 187–197, 2004.
  * [22] H. Bhargava and D. Power, “Decision support systems and web technologies: a status report,” AMCIS 2001 Proceedings, p. 46, 2001.
  * [23] R. H. Von Alan, S. T. March, J. Park, and S. Ram, “Design science in information systems research,” MIS quarterly, vol. 28, no. 1, pp. 75–105, 2004.
  * [24] J. Preece, Y. Rogers, and H. Sharp, Interaction Design: Beyond Human-Computer Interaction, 4th Edition.  Addison-Wesley Longman Ltd., 2015.
  * [25] B. Johnson and L. A. Turner, “Data collection strategies in mixed methods research,” Handbook of mixed methods in social and behavioral research, pp. 297–319, 2003.
  * [26] S. Takai and K. Ishii, “A use of subjective clustering to support affinity diagram results in customer needs analysis,” Concurrent Engineering, vol. 18, no. 2, pp. 101–109, 2010.
  * [27] P. Runeson, M. Höst, A. Rainer, and B. Regnell, “Case study research in software engineering,” in Guidelines and examples, Wiley Online Library, 2012.
  * [28] “Microsoft Forms microsoft forms is a new part of office 365 education that allows teachers and students to quickly and easily create custom quizzes, surveys, questionnaires, registrations and more..” https://flow.microsoft.com/en-us/connectors/shared_microsoftforms/microsoft-forms/.  Accessed: 2019-04-24.
  * [29] W. Foddy and W. H. Foddy, Constructing questions for interviews and questionnaires: Theory and practice in social research.  Cambridge university press, 1994.
  * [30] K. Musante and B. R. DeWalt, Participant observation: A guide for fieldworkers.  Rowman Altamira, 2010.
  * [31] C. Rivas, “Coding and analysing qualitative data,” Researching society and culture, vol. 3, pp. 367–392, 2012.
  * [32] “Affinity Diagram basic tools for process improvement.” https://www.academia.edu/27094018/Basic_Tools_for_Process_Improvement_AFFINITY_DIAGRAM_1_Module_4_AFFINITY_DIAGRAM.  Accessed: 2020-07-26.
  * [33] M. H. Blackmon, P. G. Polson, M. Kitajima, and C. Lewis, “Cognitive walkthrough for the web,” in Proceedings of the SIGCHI conference on human factors in computing systems, pp. 463–470, ACM, 2002.
  * [34] C. M. Barnum, “Usability testing essentials: Ready, set,” Test. Burlington, MA: Morgan Kaufmann, 2010.
  * [35] J. S. Dumas and J. C. Redish, “A practical guide to usability testing. norwood,” NJ: Ablex, 1993.
  * [36] C. W. Turner, J. R. Lewis, and J. Nielsen, “Determining usability test sample size,” International encyclopedia of ergonomics and human factors, vol. 3, no. 2, pp. 3084–3088, 2006.
  * [37] R. Hartson and P. S. Pyla, The UX Book: Process and guidelines for ensuring a quality user experience.  Elsevier, 2012.
  * [38] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in Advances in psychology, vol. 52, pp. 139–183, Elsevier, 1988.
  * [39] S. G. Hart, “Nasa task load index (tlx). volume 1.0; computerized version.” https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20000021488.pdf, 1986\.  Accessed: 2019-04-24.
  * [40] “vector group,” 2019.
  * [41] T. Risse and P. Knezevic, “A self-organizing data store for large scale distributed infrastructures,” in 21st International Conference on Data Engineering Workshops (ICDEW’05), pp. 1211–1211, IEEE, 2005.

##  Appendix A Interview Guides

###  A.1 Interview guide for product owner (Initial round)

Goals for the interview

  * •

Understanding the existing testing process overview.

    * –

Protocols for testing.

    * –

Testing process as a whole.

    * –

Involved stakeholders in the testing process.

  * •

Understanding the problems in the process from the perspective of the
interviewee

Questions

  1. 1.

What is your name?

  2. 2.

Which department do you work in?

  3. 3.

Which domain does your department expertise in?

  4. 4.

What can you tell about the testing process?

  5. 5.

What can you tell about the testing process?

  6. 6.

Who performs the tests in your department?

    1. (a)

What other responsibilities do they have apart from testing?

  7. 7.

We heard that the department is migrating from manual to automated testing can
you tell something about this?

  8. 8.

Would you describe the process of the existing testing as a whole which is
practiced in the department?

    1. (a)

Which stakeholder groups are involved here?

    2. (b)

Are there any typical problems with the current testing process? If yes, what
are they?

    3. (c)

Which stakeholder group suffers the most due to the problems in the testing
process?

  9. 9.

What is the role of a tester in the test process?

  10. 10.

Which tester groups should we talk to in order to know more about the work of
a tester?

###  A.2 Interview guide for testers (Initial round)

Goals for the interview

  * •

Further investigate the role of a tester in the testing process.

  * •

Find out the practices testers do during and after the test process is done

  * •

Understanding the problems in the process from the perspective of the
interviewee

Questions

  1. 1.

Would you kindly introduce yourselves?

  2. 2.

Are you all performing testing in your teams?

  3. 3.

Is the testing process common in this department?

  4. 4.

How is the test process in your perspective, as a tester?

    1. (a)

What are the objects that you test on?

    2. (b)

Do you write test-cases for the tests you do on test-objects?

    3. (c)

Can you show us examples of test-cases for test-objects?

    4. (d)

Which tools do you use for testing and why?

  5. 5.

What are the problems in the existing testing process?

  6. 6.

What do you do after the testing is done?

  7. 7.

How do you perform analysis on the test output data?

  8. 8.

Do you use any tools to perform your analysis?

###  A.3 Interview guide for Evaluation Phase

Goals for the interview

  * •

Evaluate and validate the requirements of the artifact (prototype).

  * •

Find improvements in current requirements or new additional requirements.

  * •

Investigate the data/information structure and presentation of the prototype.

Questions

  1. 1.

Did the prototype provide useful for test analysis and diagnostics?

  2. 2.

What parts in the prototype were the most useful?

  3. 3.

What part in the prototype were not useful?

  4. 4.

Is there some parts in the prototype that could be improved?

  5. 5.

Is there something that could be added to help with the test analysis and
diagnostics?

  6. 6.

Compared to the current system and process you are using for test analysis and
diagnostics, are there something missing in the prototype?

  7. 7.

How was the data structure of the data in the prototype? Described in each
specific part.

  8. 8.

Did the new data structure in the prototype help to draw conclusions/make
decisions in term of efficiency and correctness?

  9. 9.

How was the data and information presentation in the prototype? Described in
each specific part.

  10. 10.

Did the new data and information presentation in the prototype help to draw
conclusions/make decisions in term of efficiency and correctness?

  11. 11.

Is there something else you want to add?

###  A.4 Interview guide for Usability Test

Goals for the interview

  * •

Compare the two systems in regards to Effort, Efficiency, and Correctness.

  * •

Investigate the research questions:

    * –

How the data structure and presentation helped in the new system.

    * –

What aspects and factors the new system provided was an improvement.

    * –

What in the current system was causing it to be difficult.

  * •

Target research questions:

    * –

How to structure and present data to improve decision making from unstructured
data in the context of test analysis and diagnostics of automotive
electronics?

    * –

How to structure and present data in order to find the relevant information
needed to make decisions more efficiently and with reduced effort?

    * –

Which aspects in the presentation of data deteriorate decision making during
test analysis and diagnostics of automotive electronics?

####  A.4.1 Comparison of the two systems

  1. 1.

Comparing the old and the new system, how do you think the new system fares in
terms of usefulness?

  2. 2.

How was the accessibility of the required information to solve tasks in the
new system different from the old? Was it an improvement, if yes then why?

  3. 3.

While performing sub-task involving collecting the necessary files and
information for the test diagnostics using both systems, can you describe your
experience in terms of:

    1. (a)

How was the effort spent?

    2. (b)

How efficient was it?

    3. (c)

How prone to making incorrect decisions?

####  A.4.2 Questions targeting unstructured data and information presentation
[RQ1]

  1. 1.

How is the data presented that can be challenging for the test diagnostics in
the current system? In the new system?

  2. 2.

Does the information for performing test diagnostic in the current system, in
regards to DVM, log files, information about the test case, software, hardware
and stakeholder, difficult to accumulate/gather?

  3. 3.

Do you believe that having all the information easily accessible would have
made your diagnosis better? If your answer is yes in regards to:

    1. (a)

Effort needed

    2. (b)

Efficiency

    3. (c)

Correctness

  4. 4.

How did the new system do compared to the existing? If there is an improvement
can you elaborate in terms of effort and efficiency?

  5. 5.

How does the data structure in the new system compare to the old system?

    1. (a)

Does it bring any improvement? If yes, then elaborate.

    2. (b)

Does it bring any improvement in terms of effort and efficiency? If yes, which
aspects do you believe improves the effort spent and makes it more efficient?

####  A.4.3 Questions targeting decision making in terms of data presentation
and structure [RQ2]

  1. 1.

What are the factors in the current system that could make you take the wrong
conclusions or decisions? In regards to:

    1. (a)

Unstructured files and information

    2. (b)

Amount of information viewed at once

  2. 2.

Would categorization of relevant information be visually appealing while
dealing with test diagnosis?

  3. 3.

Do you think the new system improves decision making? If so, how?

    1. (a)

Do you think the presentation of the data has an influence on the improvement?

    2. (b)

Do you think the unified structure of allocating all the necessary files and
data in a central place would improve the time and effort required to take or
make decisions?

    3. (c)

Do you think that the factors you referred to are dealt with in the new
system?

####  A.4.4 Questions targeting the structural representation done in the new
system [RQ2]

  1. 1.

What is your thought on the structural representation of the data in the new
system?

  2. 2.

Was having an overview of the test case helpful?

  3. 3.

Was having information about car information or measurement configuration
helpful?

  4. 4.

Was having information about hardware/software information helpful?

  5. 5.

How important was the requirement and DVM section? Is it useful for test
diagnosis? Did it aid in decision making?

  6. 6.

How important was the historical test-case section? Is it useful for test
diagnosis? Did it aid in decision making?

  7. 7.

Was the step overview section useful for the diagnosis? Did it aid in decision
making?

  8. 8.

The steps were specifically shown in the new system. How useful was this? Did
it aid in decision making?

  9. 9.

The signals represented using graphs, how useful were they in terms of
understanding the failed signals and making decisions off it?

  10. 10.

Was the signals with comment fields to share knowledge between each other
useful?

  11. 11.

Will it be useful to add other signals to the steps directly from the new
system? Will it provide useful?

##  Appendix B Questionnaire Guide

###  B.1 Questionnaire guide for Problem Identification

  1. 1.

How much are you involved within the testing process?

    1. (a)

Tester: Prepare and conducting tests. Also, analyzing the results of the test.

    2. (b)

Analyzing: Solving the errors from the test results. Backtrack to the test
results if necessary.

    3. (c)

Both: Doing both of the above.

  2. 2.

Does your team or the other teams see the testing as an important part of the
development life-cycle? Rate them within a scale of 1-5.

  3. 3.

Questions about DVMs. Rate them within a scale of 1-5.

    1. (a)

How much is the effort of creating a DVM?

    2. (b)

Is it time consuming to create a DVM?

    3. (c)

Is the current way of creating DVM sufficient and optimal?

    4. (d)

Do you believe that having DVMs adds value to the test process?

    5. (e)

Do you think that the test sequences effectively target the outcome of the
test?

  4. 4.

Are there some challenges or limitation when DVM? (multiple choice options)

    1. (a)

Cannot express what I want to test.

    2. (b)

The process of writing DVM feels limited.

    3. (c)

The knowledge of the function is low when writing a DVM in the early phases.

    4. (d)

No, not what I can think of.

    5. (e)

Other.

  5. 5.

If have concerns about DVMs, what are the issues with them?

  6. 6.

How do you ensure the quality of the test sequence?

    1. (a)

I verify it against the DVMs Guidance.

    2. (b)

I have enough prior experience in writing test sequences.

    3. (c)

I map out requirements and verify whether the test sequence fulfill them. I
use my own methods to perform this.

    4. (d)

I use guidelines provided by the team or department.

    5. (e)

Other.

  7. 7.

Questions about the effective time in preparation of the tests. Rate them from
1-5: 1-2 hours, 3-4 hours, 5-6 hours, 7-8 hours, 9+ hours.

    1. (a)

How much effective time does it take to prepare the testing environment? i.e
software, car

    2. (b)

How much effective time does it take to conduct the actual test?

  8. 8.

What is the time consuming task during the preparation of the testing
environment? (multiple choice options)

    1. (a)

Setting up the testing software.

    2. (b)

Setting up the environment cars.

    3. (c)

Configurations (e.g software, hardware).

    4. (d)

Other.

  9. 9.

What is the time consuming task while conducting the actual test? (multiple
choice options)

    1. (a)

Waiting for it to finish.

    2. (b)

Faults related to software or hardware.

    3. (c)

Badly written DVM.

    4. (d)

Badly written requirement.

    5. (e)

Other.

  10. 10.

Questions about the post testing process. Will focus on the process after the
tests are done and when an error occurs or a fault is found in the test
results. Rate them within a scale of 1-5.

    1. (a)

How often is it because of the test sequence?

    2. (b)

How often is it because of the DVM?

    3. (c)

Is the results given from the tests sufficient to analyse and draw conclusion?

    4. (d)

Does it take a lot of time work out and analyse error or fault in the results
given from the tests?

    5. (e)

How would you rate the effort when analysing the error or fault in the results
given from the tests?

    6. (f)

When you found the source of the problem, how often is the correct stakeholder
that is contacted the right one?

  11. 11.

How much time do you usually spend working with the results given by the test
before making a conclusion?

    1. (a)

<1 hour

    2. (b)

1-2 hours

    3. (c)

3-4 hours

    4. (d)

5-6 hours

    5. (e)

Other.

  12. 12.

What are the issues or challenges when moving from manual testing to automated
testing? (multiple choice options)

    1. (a)

The functions or steps I want to test cannot be automated at the moment.

    2. (b)

It takes time to learn automated testing.

    3. (c)

It takes time to prepare necessary preparation for automated testing.

    4. (d)

I cannot backtrack the results provided from the automated testing
effectively.

    5. (e)

I prefer manual testing.

    6. (f)

Other.

  13. 13.

What kind of issues do the test data, log files and results from the automated
testing have? (multiple choice options)

    1. (a)

The large amount of data.

    2. (b)

The large amount of irrelevant data.

    3. (c)

It is hard to find the relevant data.

    4. (d)

Missing information or data.

    5. (e)

The overall structure of how the test data and log files are stored.

    6. (f)

The data/log files are not in the same place.

    7. (g)

Lacking better visualization.

    8. (h)

Difficult to map the test case(DVM) with the output of the data.

    9. (i)

The files related to the test-object are stored in various places, making it
difficult to collectively analyze them.

    10. (j)

Other.

  14. 14.

Which software do you use for analyzing the results of the tests? (multiple
choice options)

    1. (a)

CANalyzer

    2. (b)

CANape

    3. (c)

Other.

  15. 15.

Is there some other problems or challenges you want to add?

  16. 16.

How long have you been in your current position in the department?

    1. (a)

0-1 years

    2. (b)

1-2 years

    3. (c)

3-4 years

    4. (d)

5+ years

  17. 17.

Demographic: Age

    1. (a)

<20

    2. (b)

21-30

    3. (c)

31-40

    4. (d)

41-50

    5. (e)

50+

  18. 18.

Demography: Nationality

###  B.2 Questionnaire guide for gathering subjects for Usability Test

  1. 1.

Name

  2. 2.

Email

  3. 3.

Department and Team

  4. 4.

Experience in the field of testing (in years)

##  Appendix C Cognitive Walk-through Tasks

Tasks for the participants to perform for the cognitive walk-through. The
tasks are made for the participants to find the necessary information and data
in the prototype that could be helpful during test analysis and diagnostics.

###  C.1 Tasks for Low-Fidelity Prototype

  1. 1.

Get information about the test case.

    1. (a)

Open the system.

    2. (b)

Go to the Overview.

    3. (c)

Get general information.

    4. (d)

Get Stakeholder and Tester information.

    5. (e)

Get hardware and software information.

  2. 2.

Get information about the requirement and DVM of the test case.

    1. (a)

Open the system.

    2. (b)

Go to DVM.

    3. (c)

Get description.

    4. (d)

Get test sequence.

  3. 3.

Get information about a specific step.

    1. (a)

Open the system.

    2. (b)

Go to a step number.

    3. (c)

Get the sensor signals involved in that specific step.

    4. (d)

Read the description of the specific step.

    5. (e)

Compare the sensor signals outputs in the graph.

  4. 4.

Get information about test cases which are similar with the current test case.

    1. (a)

Open the system.

    2. (b)

Go to Historical Data.

    3. (c)

Get the list of similar test cases.

###  C.2 Tasks for High-Fidelity Prototype

  1. 1.

Get an overview of the test case.

    1. (a)

Open the system.

    2. (b)

Go to Overview.

    3. (c)

Get the results of the test case.

    4. (d)

Get an overview of the steps in the test case.

    5. (e)

Get the sensor signals used in the test case.

    6. (f)

Get the buses involved in the test case.

  2. 2.

Get information about the requirement and DVM of the test case.

    1. (a)

Open the system.

    2. (b)

Go to the DVM.

    3. (c)

Get description.

    4. (d)

Get test sequence.

  3. 3.

Get information about test cases which are similar with the current test case.

    1. (a)

Open the system.

    2. (b)

Go to Historical Data.

    3. (c)

Get the list of similar test cases.

  4. 4.

Get the information related to the test case.

    1. (a)

Open the system.

    2. (b)

Get the software and hardware information.

    3. (c)

Get the car information, such as database and configurations.

    4. (d)

Get the tester information.

    5. (e)

Get the stakeholder information.

  5. 5.

Get the links to the file-repository of the log-files used in the test case.

    1. (a)

Open the system.

    2. (b)

Go to link to log files.

  6. 6.

Get information about a specific step.

    1. (a)

Open the system.

    2. (b)

Go to a step number.

    3. (c)

Get the sensor signals involved in that specific step.

    4. (d)

Read the description of the specific step.

    5. (e)

Compare the sensor signals outputs in the graph.

##  Appendix D Coded Interviews

Figure D.1: Findings from the interview with product-owner.

Figure D.2: Findings from the interview with testers group 1.

Figure D.3: Findings from the interview with former testers.

##  Appendix E Requirements

The user requirement specifications, including description, rationale,
dependencies and use case can be seen below. The motivation for the
requirements and how it would be useful for test analysis were described in
the rationale. Use case described how the user could use the feature of the
requirement in the context of a web-based system (Dashboard) which was the
artifact.

UR1 |  View an overview of the test case.
---|---
Description |  User can view an overview of the test case. The overview page of the test case was the landing page of Dashboard and that included:  • The results of the test (i.e if it failed or passed) • Overview of each step in the test case • Overview of signals involved in the test case • Overview of the buses involved in the test case • Time and dates for the test case (Time-stamp)
Rationale |  This feature lets the user understand and get brief information on the test case and its test object. By getting this overview, it will help the user to quickly navigate to more interesting and relevant aspects/information during the analysis of the test object.
Dependencies |  UR3, UR10
Use Case |  1. Open Dashboard. 2. View the overview of the test object: • The overview page was the landing page of the Dashboard, no more action needed. • If already browsing the Dashboard, click on the “Overview” tab on the sidebar.
UR2 |  View and download (if available) the initiation and metadata files related to the test object and the test case.
---|---
Description |  User can see the location of initiation and metadata files containing the data and information related to the test object and the test case. It also includes the directory, identification, or name of the files making it easier to locate them if required.

The data and information presented in the Dashboard used these files as a
reference to present to the most relevant context for the test case. Not
everything presented in the Dashboard had access to those files, but a
reference link to where the information was stored was indicated regardless.
Rationale |  This feature lets the user access all the initiation and metadata files that are associated with the test-object. Hence it also allowed the users to get the files quickly for initializing their test suites. They were required to perform the test again or open log files in their preferred analysis tools for a more in-depth analysis, which was necessary for more advanced test cases.

Providing faster and direct access to the user with the initiation and
metadata files would increase the correctness, reduce effort, and increase the
test process efficiency and analysis compared to finding the files manually.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click the “Links” on the sidebar.
UR3 |  View information on a particular step in the test case.
---|---
Description |  System should map test sequence with sensor signal at particular timestamp and see the information of what happened at that timestamp. The information that will be included are as:  • The expected values of the signals and result in the step. • The signals that are being tested in the step. • Graphs for the signals and the timestamp for the step. • Being able to comment on the signals, for knowledge sharing and archiving. • Marking the frequently analyzed/investigated signal as "Favorite",
Rationale |  This feature will give the user a brief overview of what happen during each step of the test sequence during the testing. The signals showed here will be analyzed here by comparing the result with the expected outputs, as well as between other signals.

The goal of this feature is to provide a brief overview of a particular step,
to allow the user to visualize the relevant and correct information regarding
the sensor signals for that step. Information gathered here, could be used for
quickly performing test analysis for preferably medium difficulty level tests.
For more in-depth analysis, other testing tools could be used instead.
Dependencies |  UR4, UR5, UR14, UR15
Use Case |  1. Open Dashboard. 2. Click on: • “Step Overview” on the sidebar to get all the steps. • “Step [number]” on the sidebar to get a specific step.
UR4 |  View graphs of the sensor signals in the test case.
---|---
Description |  User can visualize the graphs of the sensor signals, generated from test output. The graph included the sensor signals values/outputs in the y-axis and timestamp in the x-axis.
Rationale |  This feature will provide the user with better visualization of the signals and the steps, which would help with analysis by getting a better overview of each step. By displaying the sensor signals associated with the test case, this would increase the efficiency and correctness. Furthermore, the users will not be overwhelmed by witnessing many sensor signal data at once due to how they were displayed according to steps. Overall, this feature provided the users with a simple graph that could be useful for simple analysis by comparing signals output values, tailoring towards analyzing test cases with medium difficulty level. More in-depth analysis could be done on other analysis programs if required.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click on: • “Step Overview” on the sidebar to get all the steps. • “Step [number]” on the sidebar to get a specific step.
UR5 |  Search for signals.
---|---
Description |  User could search for the signals in the test output for analysis.
Rationale |  This feature would let users attach and analyze additional sensor signal outputs that were not the default ones provided by for each step. This would help the users see correlations and perform analysis directly in the Dashboard.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click on: • “Step Overview” on the sidebar to get all the steps. • “Step [number]” on the sidebar to get a specific step.
UR6 |  View the stakeholder for the test object.
---|---
Description |  User could see the responsible stakeholder for a test object.
Rationale |  This feature would help the user to see the stakeholders who were responsible for test object, which would make it easier to contact them if fixing an error/issue found in the test was their responsibility or ask domain specific question about the test case/object.

Providing a list of the stakeholders and their roles, would help the users
decide which stakeholder to contact efficiently and reduce the efforts
compared to manually finding the correct stakeholder.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click on the “Stakeholder” in the sidebar: • More information about the stakeholder by clicking the “i” icon next to the stakeholder.
UR7 |  View information of the test performers.
---|---
Description |  User could see the information of the test performers.
Rationale |  This feature would help the users to see the testers who conducted the tests, which would make it easier to contact them for more information or questions about test case or object.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click on the “Stakeholder” in the sidebar: • More information about the tester by clicking the “i” icon next to the testers.
UR8 |  View the Design Verification Method (DVM) for the test object.
---|---
Description |  Users will see the DVM for the test object. DVM includes information about the pre-condition, post-condition, test requirements, and test object, and test case description. The DVM name and revision version are also included for the users to avoid making mistakes, like refereeing the wrong DVM version for test analysis.

The user will also see the test sequence, which describes how the test was
supposed to be conducted step by step and its verdicts, including expected and
actual results.
Rationale |  This feature will be used to verify test-cases with the DVM by visually interpreting it. By providing this feature, users will have an easier and faster access to the correct DVM.
Dependencies |  -
Use Case |  1. Open Dashboard: • DVM: Click “DVM” on the sidebar. • Test Sequence: Click “Test Sequence” on the sidebar.
UR9 |  View the requirements for the test object.
---|---
Description |  User will be able to see the requirements for the test object. Requirements includes description of the test object’s constraints, expected outputs, limitations and objects(e.g signals/ECU) involved. The requirement and revision version is also included.
Rationale |  This feature is used to verify test-cases with the feature requirement by visually interpreting it. By providing this feature, users will have an easier and faster access to the correct requirement.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click “Requirement” on the sidebar.
UR10 |  View the ECUs and its associated Bus of the test object used in the test case.
---|---
Description |  User will be able to see a list of ECUs and its associated Bus of the test object used in the test case. Name of the ECUs and Buses are included to allow users to quickly identify them as opposed to doing it manually.
Rationale |  This feature allows the user to get an overview of the context for the test case to be able to see what kind ECUs and buses are involved in the testing. This feature will help the user be more efficient when locating relevant data and information during analysis in regards to buses and ECUs.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. View the overview of the test object: • The overview page is the landing page of the Dashboard, no more action needed. • If already browsing the Dashboard, click on the “Overview” tab on the sidebar.
UR11 |  View the details of the software and hardware read-outs.
---|---
Description |  User will be able to see the software and hardware information of the test object. The read-outs are generated by the testing tools and provide useful information to the user for test analysis. The read-out specific information were:  • Identifiers • Response outputs • ECU (serial number and part number) • Timestamps • Request and response time
Rationale |  This feature gives the user the relevant read-outs of software and hardware information. As the read-outs usually contains a large amount of information, providing them briefly and concisely here will help to users reduce the workload when trying to understand and find read-out specific information.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click “Information” on the sidebar.
UR12 |  View the the calibration and configuration of the test object for the test case.
---|---
Description |  User will be able to see the calibration and configuration information of the test when it was conducted. The calibration is needed by the test suite to let users perform/re-produce tests. The calibration information included:  • Topology: Car ID and Car Family ID • ECU name and version • Bus name and version • Database name and version for signal sensor signals • Measurement and analysis tools configuration
Rationale |  This feature let the user to set-up the calibration and configuration for more in-depth analysis, using their preferred analysis tool more efficient. Also, by providing the correct calibration and configuration data, users will not require to manually look for them in different storages. Furthermore, this feature will increase the correctness during the test analysis and diagnostics as testers usually had to do trail and error to validate correct calibration and configuration files amongst the incorrect ones.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click “Information” on the sidebar.
UR13 |  View historical data for a test object.
---|---
Description |  User can view previous test related data for similar test object. The similarity of a test object is determined by the test object, test case, requirement and DVM. The metadata displayed for the user are:  • The verdict (i.e success or failed) • Name of the test case • Date and time • Domain • The author who conducted the test • Link to the directory to initiation and metadata files off the test-case
Rationale |  This feature will let the user backtrack the previous test results of similar test cases/test-objects to get references and knowledge for their analysis.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click “Historical Data” on the sidebar.
UR14 |  View and add favorite signals.
---|---
Description |  User can favorite signals to store them locally to access them more easily.
Rationale |  This feature lets the user get their most frequently used sensor signals faster and more efficiently. In some cases, some signals should always be checked during the analysis.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click on: • “Step Overview” on the sidebar to get all the steps. • “Step [number]” on the sidebar to get a specific step. 3. A heart button next to the signals: • Click a hollow heart to like. • Click a filled heart to unlike.
UR15 |  View and add comments on signals.
---|---
Description |  Users can write and read comments on signals.
Rationale |  This feature allows knowledge sharing between users. It works as a forum to discuss issues.
Dependencies |  -
Use Case |  1. Open Dashboard. 2. Click on: • “Step Overview” on the sidebar to get all the steps. • “Step [number]” on the sidebar to get a specific step. 3. Click on the comment icon next to the sensor signal, a pop-up windows of comments will be shown. 4. Action: • Read comments in chronological order. • Write a comment and then post it.

##  Appendix F Low-fidelity Prototype

Figure F.1: Low-fidelity prototype sketch drawn on whiteboard.

##  Appendix G High-fidelity Prototype

Figure G.1: High-fidelity prototype of overview view.

Figure G.2: High-fidelity prototype of DVM view.

Figure G.3: High-fidelity prototype of historical view.

Figure G.4: High-fidelity prototype of information view.

Figure G.5: High-fidelity prototype of step view.

Figure G.6: High-fidelity prototype of step sub-views.

##  Appendix H Final High-fidelity Prototype

Figure H.1: Final high-fidelity prototype of overview view.

Figure H.2: Final high-fidelity prototype of requirement view.

Figure H.3: Final high-fidelity prototype of DVM view.

Figure H.4: Final high-fidelity prototype of test sequence view.

Figure H.5: Final high-fidelity prototype of historical view.

Figure H.6: Final high-fidelity prototype of information view.

Figure H.7: Final high-fidelity prototype of links view.

Figure H.8: Final high-fidelity prototype of stakeholder view.

Figure H.9: Final high-fidelity prototype of stakeholder sub-view.

Figure H.10: Final high-fidelity prototype of step overview view.

Figure H.11: Final high-fidelity prototype of step signal comment view.

Figure H.12: Final high-fidelity prototype of step signal search and add view.

Figure H.13: Final high-fidelity prototype of specific step view.

Figure H.14: Final high-fidelity prototype of specific step view expanded.

##  Appendix I NASA Task Load Index Results for Current System / Old System

NASA Task Load Index (TLX) result for each task collected from the usability
test performed on the current system. Paper and pen approach was used with a
TLX
sheet111https://humansystems.arc.nasa.gov/groups/TLX/downloads/TLXScale.pdf
providing a 1-21 scale, which were multiple with 5 to give a 0-100% scale.
Weight for each demand was given by comparing each demand with each other
where the subject determined which one were the more important one.

Tester 1
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 60 | 75 | 85 | 10 | 5
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 25 | 85 | 75 | 55 | 2
Performance | 10 | 75 | 25 | 55 | 3
Effort | 30 | 70 | 75 | 40 | 3
Frustration | 20 | 70 | 60 | 40 | 2
Product Sum | 510 | 1120 | 995 | 525 | -
Score | 34 | 74 | 66 | 35 | -
Tester 2
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 40 | 70 | 60 | 70 | 4
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 45 | 60 | 65 | 65 | 2
Performance | 15 | 65 | 30 | 20 | 1
Effort | 70 | 80 | 65 | 40 | 5
Frustration | 60 | 85 | 35 | 60 | 3
Product Sum | 795 | 1120 | 830 | 810 | -
Score | 53 | 74 | 55 | 54 | -
Tester 3
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 65 | 70 | 40 | 75 | 4
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 50 | 50 | 50 | 40 | 3
Performance | 20 | 25 | 20 | 40 | 2
Effort | 35 | 50 | 25 | 50 | 3
Frustration | 25 | 65 | 30 | 30 | 3
Product Sum | 630 | 825 | 515 | 740 | -
Score | 42 | 55 | 34 | 49 | -
Tester 4
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 65 | 20 | 80 | 70 | 4
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 35 | 20 | 75 | 75 | 3
Performance | 30 | 25 | 15 | 10 | 1
Effort | 75 | 50 | 75 | 75 | 4
Frustration | 50 | 30 | 60 | 70 | 3
Product Sum | 845 | 455 | 1040 | 1025 | -
Score | 56 | 30 | 69 | 68 | -
Tester 5
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 80 | 60 | 20 | 45 | 5
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 40 | 20 | 25 | 25 | 1
Performance | 20 | 25 | 15 | 30 | 2
Effort | 50 | 30 | 20 | 25 | 4
Frustration | 20 | 30 | 15 | 55 | 3
Product Sum | 740 | 580 | 280 | 575 | -
Score | 49 | 38 | 18 | 38 | -
Tester 6
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 55 | 60 | 55 | 45 | 4
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 40 | 35 | 70 | 55 | 2
Performance | 20 | 20 | 40 | 50 | 3
Effort | 65 | 45 | 50 | 60 | 3
Frustration | 40 | 65 | 45 | 45 | 3
Product Sum | 675 | 700 | 765 | 765 | -
Score | 45 | 46 | 51 | 51 | -

##  Appendix J NASA Task Load Index Results for New System

NASA Task Load Index (TLX) result for each task collected from the usability
test performed on the new system. Paper and pen approach was used with a TLX
sheet111https://humansystems.arc.nasa.gov/groups/TLX/downloads/TLXScale.pdf
providing a 1-21 scale, which were multiple with 5 to give a 0-100% scale.
Weight for each demand was given by comparing each demand with each other
where the subject determined which one were the more important one.

Tester 1
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 30 | 5 | 65 | 10 | 3
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 75 | 15 | 65 | 35 | 4
Performance | 25 | 20 | 15 | 5 | 2
Effort | 40 | 10 | 70 | 10 | 3
Frustration | 60 | 20 | 30 | 25 | 3
Product Sum | 740 | 205 | 785 | 285 | -
Score | 49 | 13 | 52 | 19 | -
Tester 2
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 20 | 10 | 30 | 45 | 4
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 25 | 15 | 30 | 45 | 2
Performance | 15 | 5 | 15 | 15 | 1
Effort | 20 | 15 | 50 | 15 | 4
Frustration | 15 | 15 | 40 | 40 | 4
Product Sum | 285 | 195 | 555 | 505 | -
Score | 19 | 13 | 37 | 33 | -
Tester 3
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 10 | 15 | 30 | 20 | 5
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 10 | 15 | 35 | 20 | 2
Performance | 10 | 15 | 15 | 20 | 2
Effort | 20 | 15 | 25 | 20 | 3
Frustration | 15 | 10 | 15 | 20 | 3
Product Sum | 195 | 210 | 370 | 300 | -
Score | 13 | 14 | 24 | 20 | -
Tester 4
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 30 | 20 | 70 | 35 | 4
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 20 | 20 | 35 | 20 | 3
Performance | 15 | 25 | 35 | 10 | 2
Effort | 40 | 35 | 75 | 25 | 3
Frustration | 15 | 15 | 50 | 20 | 3
Product Sum | 375 | 340 | 830 | 355 | -
Score | 25 | 22 | 55 | 23 | -
Tester 5
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 10 | 25 | 20 | 20 | 3
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 25 | 20 | 25 | 20 | 4
Performance | 10 | 25 | 15 | 15 | 2
Effort | 15 | 15 | 15 | 10 | 2
Frustration | 10 | 20 | 15 | 35 | 4
Product Sum | 220 | 315 | 280 | 330 | -
Score | 14 | 21 | 18 | 22 | -
Tester 6
---
Demands | Task 1 | Task 2 | Task 3 | Task 4 | Weight
Mental Demand | 15 | 20 | 40 | 30 | 4
Physical Demand | 5 | 5 | 5 | 5 | 0
Temporal Demand | 15 | 15 | 30 | 15 | 4
Performance | 15 | 20 | 40 | 10 | 3
Effort | 25 | 20 | 50 | 15 | 2
Frustration | 10 | 15 | 45 | 20 | 2
Product Sum | 235 | 270 | 590 | 280 | -
Score | 15 | 18 | 39 | 18 | -

##  Appendix K Coded Usability Test Interviews

Figure K.1: Findings from the usability test.