Paper Title: LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows.



: Application-agnostic Integration of Large Language Models]
: Application-agnostic Integration of Large 
 Language Models to Support Personal Writing Workflows





  Department of Computer Science
  ETH Zürich
  Switzerland


firstname.lastname@inf.ethz.ch



§ ABSTRACT 
To enhance productivity and to streamline workflows, there is a growing trend to embed large language model (LLM) functionality into applications, from browser-based web apps to native apps that run on personal computers.
Here, we introduce LLM-for-X, a system-wide shortcut layer that seamlessly augments any application with LLM services through a lightweight popup dialog.
Our native layer seamlessly connects front-end applications to popular LLM backends, such as ChatGPT and Gemini, using their uniform chat front-ends as the programming interface or their custom API calls.
We demonstrate the benefits of LLM-for-X across a wide variety of applications, including Microsoft Office, VSCode, and Adobe Acrobat as well as popular web apps such as Overleaf.
In our evaluation, we compared LLM-for-X with ChatGPT's web interface in a series of tasks, showing that our approach can provide users with quick, efficient, and easy-to-use LLM assistance without context switching to support writing and reading tasks that is agnostic of the specific application.





<ccs2012>
   <concept>
       <concept_id>10003120.10003121.10003124</concept_id>
       <concept_desc>Human-centered computing Interaction paradigms</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10003120.10003121.10003128.10011753</concept_id>
       <concept_desc>Human-centered computing Text input</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
   <concept>
       <concept_id>10003120.10003121.10003122.10003334</concept_id>
       <concept_desc>Human-centered computing User studies</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Human-centered computing Interaction paradigms
[300]Human-centered computing Text input
[300]Human-centered computing User studies




[
    Lukas Teufelberger, Xintong Liu, Zhipeng Li, https://orcid.org/0000-0003-3414-7142Max Moebus, and https://orcid.org/0000-0001-9655-9519Christian Holz
    August 5, 2024
=========================================================================================================================================================





§ INTRODUCTION


Large language models (LLMs) are becoming a common part of authoring processes, such as writing <cit.>, editing <cit.>, and question answering <cit.>.
Users across different fields now employ LLMs to enhance productivity <cit.> and creativity <cit.>.
LLM-based co-authoring and co-editing thus mark a shift in how people approach content creation <cit.> and how they perceive individual agency over the process <cit.>.


The dominant mode of interaction with LLMs is chat interfaces in the form of question & answer dialogs.
Chat assistants offered by the major providers of LLM backend services, such as OpenAI ChatGPT <cit.>, Google Gemini <cit.>, or Anthropic Claude <cit.> mimic human conversations to make interactions with LLMs natural.
Via chat queries, users can retrieve information, conduct research, and solve their problems <cit.>.
These capabilities have made chat-based LLMs a promising alternative to conventional search interfaces <cit.> to support users throughout task executions.




Beyond question & answer interaction, users frequently copy & paste content into and from these chat interfaces to effectively execute a variety of tasks inside other applications.
This workflow uses the clipboard as the key interface to transfer textual information back and forth across tabs, windows, and applications.
The latest generation of LLMs also supports receiving images as input <cit.> and generating images as output <cit.>, which augments the chat interface with a rich modality for interaction.

Following the success, many developers have incorporated LLM services into their products, from smaller apps (e.g., Grammarly <cit.>, personal AI Pi by Inflection <cit.>, My AI by Snapchat <cit.>) to large software suites (e.g., Copilot in Microsoft Office <cit.>, Google Workspace <cit.>, Amazon Lex <cit.>), often charging (separately) for this added functionality.
As a consequence, users must join multiple subscriptions, one for each app to enjoy LLM-based services—despite the large overlap of their capabilities and in the face of free general purpose use <cit.>.
For this reason, copy & paste may remain the dominant interface for users to bridge LLM assistance and the apps they use to complete tasks in a desktop environment.

We introduce , a user interface technique that provides interaction with LLM backends to users within any frontend app.
As shown in Figure <ref>, requires no copy & paste or switching windows to LLM frontends, and it allows users to almost directly operate on text inside an app using a light-weight on-demand popup UI.
Our system service connects text selections and user queries to an LLM by either emulating user input to a chat interface (e.g., an existing subscription for ChatGPT) and seamlessly transfers responses back into the app.
Alternatively, connects to LLM backend APIs to retrieve responses.
In both cases, our approach minimizes the effort to interface wiht LLMs, allowing users to focus on a task within an app without context switching.



 §.§ Walk-through and interaction design


is designed for efficient keyboard use, supporting the writing and editing process of text across apps with shortcuts.
Figure <ref> shows a typical app scenario.
The user selects text inside an app, here Overleaf running in the web browser, and triggers via keyboard shortcut ().
A prompt menu appears with several options for quick actions and associated prompts that can be triggered using the number keys.
Alternatively, the user may enter a custom LLM query.
Here, the user types “fix spelling mistakes”, which sends the prompt and the text selection to an LLM chat in the background.
then previews the LLM response in the menu, which the user inserts into Overleaf by pressing .
This closes the menu and completes the interaction flow.

Below, we describe more interaction scenarios.
We also refer the reader to our video figure for additional demonstrations: 

.







  §.§.§ Iterating on LLM responses


Before inserting the LLM response, allows users to refine the prompt to iterate on the generated responses (Figure <ref>a).
Here, the dialog shows the original text and the LLM response side-by-side, highlighting changes using a diff view.
Users may alter the prompt, resubmit the query on the selected text, and see the updated LLM response in the preview area.
Our tool thereby maintains the chat session in the background, such that query refinements preserve previous output in the LLM chat interface for context.




  §.§.§ Insert vs. replace text in the app


While text replacement may be the most common action, additionally supports appending the LLM response below the text selection when the user holds a modifier key  before pressing  for insertion (Figure <ref>b).
This functionality can be useful when the purpose of the selected text is to act as specific context to inform the completion of the task.
Examples include coding tasks where instructions or descriptions such as those in comments above a function signature should be left intact.



  §.§.§ Additional context to the LLM


Our tool is able to, if selected by the user, provide additional context to the LLM query by including the textual content surrounding the selection.
The additional context is flagged as context in 's prompt to the LLM's chat interface to ensure that the response does not directly include part of this augmentation.

In any case, extracts the current app's name and the window title as context for prompt customization.
We also use the window title to decide between resuming a previous LLM chat session or starting a new session.



  §.§.§ Direct in-place response


When the user holds the modifier  while submitting a query, the prompt menu will close immediately and our system service will insert the LLM response directly in the foreground app by simulating key presses—letter by letter and in real-time as produced by the chat interface (Figure <ref>c).
Because this insertion appears to the foreground app as entered by the user, the LLM response can be handled as such and, for example, be undone using the app's regular `undo' function.



  §.§.§ Querying information without inserting responses


Finally, can also be used to simply obtain additional information.
Selecting the relevant text, pressing , and hitting  (for `explain') or  (for `translate') will produce the LLM's chat response in the menu's preview area as before.
This functionality can be useful for contextual information, such as in a reading app or PDF viewer when users may require context (Figure <ref>d).
Another example is the use of the tool for simultaneous translation (Figure <ref>d) and explanation on foreign language (news) websites. 

Since the app in focus may not support text entry, such as in the case of the PDF viewer, recognizes this and hides the  button.
Users may still refine the query for further details.



 §.§ Design objectives


 We primarily designed to allow for quick interaction with LLM backends without the need for context (or app) switching as shown in Figure <ref>.
While our interaction design is optimized for keyboard use (e.g., shortcuts to summon via , to select menu items via 1..5,  for pasting, and modifier keys), all options are equally accessible via mouse use, too.
Inside native apps, 's prompt menu can be invoked using an icon in the system tray, whereas a toolbar icon triggers it in the browser.
The menu stays open and in front until the user presses  or clicks the button in the top right. 
Upon pressing , inserts or replaces the selection at the location of the cursor.
Our menu will remain active even if users click a different position in the text editor, such that they can determine where to paste the LLM response before pressing  or  to close the menu.

The visual design if 's popup is lightweight to minimize distraction from the foreground app.
's UI has three main elements: query input field (which is in focus when the menu appears), suggested actions (chosen based on popular use-cases of ChatGPT <cit.>, accessible via shortcuts), and the preview output field for LLM responses, where changes are color-highlighted.
The latter can be navigated with / or the scrollbar, while typing will modify the query prompt.









 §.§ Contributions


Our research makes two main contributions:




  * , an operating system-wide shortcut interaction technique that allows prompting LLM backends from within any native app or web app, optionally based on a text selection for manipulation.
Our technique is implemented through an OS service and can alternatively interface with a browser extension to facilitate LLM communication or an LLM API backend directly, either offered by popular LLM providers such as OpenAI ChatGPT and Google Gemini.
Our technique thus bridges their text-based functionality with text input inside any app.


  * a user study with 14 participants to compare their performance during writing, editing, and coding tasks using and .
Participants were significantly faster in editing text and reported higher usability (using SUS <cit.>), as allowed them to stay within one app without context switching.





§ RELATED WORK


is related to virtual assistants, LLM-based support tools, and shortcut interaction.



 §.§ Virtual Assistants


A virtual assistant (VA) is a software agent that carries out a variety of tasks or services in response to user input, such as commands, questions, and verbal prompts <cit.>. 
 categorize the research of intelligent writing assistants into task, user, technology, interaction, and ecosystems <cit.>.
falls under the interaction aspect, specifically emphasizing the hybrid interaction metaphor, localized user interface, and explicit user control. 



  §.§.§ Intelligent Personal Assistants

With the integration of machine learning and advanced speech recognition technologies into personal devices, the prevalence of intelligent personal assistants (IPAs) highlights the effectiveness of AI-powered personal assistants in everyday tasks <cit.>. 
These software agents are present in wearables, smartphones, and desktops <cit.>, such as Siri <cit.>, Alexa <cit.>, and Cortana <cit.>.
However, most users rely on IPAs for only basic tasks, such as playing music and scheduling appointments <cit.>.



  §.§.§ IPAs in the Age of LLMs

Researchers envision the next generation of IPAs to be powered by LLMs and to become more prevalent as omnipresent personal companions <cit.>. 
Early prototypes for LLM-based IPAs on smartphones include GPTVoiceTasker <cit.> and GPTDroid <cit.>. 
Windows Copilot <cit.> is among the first commercially-available personal companions for desktop environments. 






 §.§ LLM-based Tools

With the rapid expansion of open-source NLP research, there are many studies on specialized bots and apps for specific use cases such as domain-specific or task-orientated text generation <cit.>. 
Popular uses for LLMs include text generation, information retrieval, and problem solving <cit.>. 



  §.§.§ Writing

Co-writing systems such as Wordcraft <cit.> show that users make use of AI assistants throughout many stages in the writing process <cit.> and tools such as or TexGPT from Writeful <cit.> even offer platforms-specific AI assistants, e.g., for Overleaf.
More advanced writing assistants focus on providing new perspectives for reflection and inspiration in addition to grammar correction and text polishing <cit.>. 



  §.§.§ Information Retrieval

Learning and browsing online could also benefit from assistance.
Complex search tasks require similarly complex search prompts <cit.>.
The success of complex search tasks is then heavily dependent on the quality of the search prompt <cit.>.
GPT models also possess suitable translation capabilities <cit.> and can be useful in educational settings <cit.>.



  §.§.§ Coding

AI assistants are also popular for aiding software engineering tasks as so-called copilots or companions <cit.>.
The advancements of LLMs fine-tuned on publicly available code facilitate more efficient coding with AI assistants (e.g., Codex <cit.>).
Previous studies showed that while Github Copilot and ChatGPT-assisted coding does not outperform human-written solutions, users acknowledge their ability to handle fundamental tasks <cit.>.




 §.§ Efficient interaction and shortcuts

When trying to harness the full power of LLMs and AI assistants, the bottleneck during interaction with generative AI models is speed and output quality <cit.>.
Since many LLM commands are invoked through menus, keyboards can provide preferable and faster access over mouse use <cit.> via shortcuts.

Further, using AI assistants to everyday tasks should disrupt the underlying tasks as little as possible <cit.>.
Across a broad range of professions including primary care <cit.> and software engineering <cit.>, such interruptions otherwise significantly impair productivity and quality of work, even those as small as context switching.
For software engineers, context switching strongly affects perceived productivity and mood <cit.>.
AI assistants should thus be seamlessly integrated into everyday tasks to minimize interruptions and context switches.



§ IMPLEMENTATION


Figure <ref> shows an overview of the architecture of our implementation, including 's operating system (OS)-level background service, which produces its prompt menu UI, its browser extension, and its optional direct API use.
's prompt menu is the main interactive and user-facing element, configured by our background service that monitors for short-cut activation.
This service also interfaces with the content inside native apps and retrieves surrounding content for context.
For interfacing with web apps, our system includes a browser extension for deep access to web pages.
In both cases, our background service retrieves selected text and surrounding content and replaces or inserts selections with the responses from the LLM.
Our background service also interfaces with the web-based frontend of LLM services via our browser extension and operates them by simulating a user's copy&paste interaction or alternatively directly queries backends via their APIs.

We implemented to run on Windows 10 and up.
It supports Google Chrome or Microsoft Edge through its extension API.
If support for native apps is not needed, can also operate solely based on its browser extension (including an in-browser short-cut listener and a browser-based menu).
This makes it compatible with all platforms that fully support Chrome or Edge and extensions without requiring a native component.




 §.§ OS-level background service


runs as a background app to listen for triggers of the prompt menu across applications and web browser tabs.
We implemented the service is C# using .net APIs, registering a system-level keyboard hook to watch for global keyboard shortcuts.
Once our service detects a shortcut, our native app interface extracts application details and text selection to prepare the prompt.




 §.§ Native app interface (via accessibility API)


Access to text contents within an app is available via UI Automation API (UIA) on Windows, which is part of the Microsoft Accessibility API.
Our service has a registry key in Windows set to be able to communicate with the browser. 
From the currently focused element, we obtain the window title and Process ID (PID), the latter we use to capture the current app's name.
Both window title and app name we use for context. 

To extract selected text, we start with the focused element and browse the subtree of elements for a UI element with a UIA's .
Once the property is available, we discover the content for the prompt body via UIA's .

can also obtain surrounding text for additional context as mentioned above.
We reuse the identifier of the UI element with the  and obtain the entire body of the text.
While we use the selected text as main input into the prompt, the rest we add to the prompt for context explicitly declared as such.



  §.§.§ Paste LLM responses

Finally, to insert the responses from the chat interface, we emulate direct user input into the control.
's backend first copies the response to the clipboard and then invokes `Paste' with  to simulate the paste operation.
The benefit of proceeding this way is that input gets entered into the app's undo/redo stack.
In case the text is not to be replaced but inserted below, we simulate a `Cursor Right' key press beforehand.




  §.§.§ Fallback: if the API fails


While the UI Automation API supports most apps on Windows, we have found some not to reveal text content or selections.
In case discovers no existence of , our service makes use of the clipboard and injects a copy shortcut command ().

Likewise, to retrieve the surrounding context if so configured, we simulate `Select All' () and another copy shortcut.
Just before, we simulate pressing `Cursor Right' and `Space' to enter an operation to the undo stack.
After selecting all and copying the text, we undo the last command (space) to set the cursor position to the bottom of the selection.
We cannot recover the text selection state, however.




 §.§ Web app interface (via browser extension)


If a web browser is in the foreground, 's browser extension connects to our backend service upon detecting the trigger shortcut.
The extension then interfaces with the foreground web page to extract the selected text (if any), textual content, and later handle text replacement or insertion.

Through Javascript and the DOM API, 's browser extension obtains the selected element in the DOM and extracts the selected text inside.
Our extension then finds the first common ancestor and takes its  to extract the selection's context.



  §.§.§ Insert LLM responses

's extension again uses the current selection and appends  to the end. 
This reliably works for both editable UI elements (e.g.,  and ) but also any other elements (e.g.,  with  set).




  §.§.§ If responses cannot be pasted


Finally, our extension determines whether the element that contains the selected text is not editable, such that an LLM response could not be pasted.
In this case, 's prompt menu does not show the  button.
We conclude this from the element's tag name (input or textarea) and from whether it is set to read-only, disabled, or if it is .



 §.§ Native service-to-extension interface


For communication between our browser extension and native service, we send commands via the Native Messaging API <cit.>.
Our browser extension thereby starts the native app and uses the standard input/output interface for message exchange.



 §.§ LLM interface (via browser extension)

currently supports ChatGPT, Mistral, and Gemini.
Because they all follow the same interaction paradigm using a chat interface, our browser extension emulates user input when a query is submitted from the prompt menu, extracts the response from the LLM web UI, and transfers it back to the prompt menu.




  §.§.§ Communicating with LLM chat interface pages


Upon opening an LLM provider's chat interface, our extension detects the chat window, the query input field, and the response element.
The extension searches for the input element using a combination of tag names and element paths.
It then inserts the query by either setting the  or  of the element with dispatching a few events first to simulate user interaction.
The query is submitted by injecting a `return' or `click' event.

Afterwards, our extension repeatedly polls the DOM in short intervals to detect changes in the output element.
We obtain the text from the  property of the output element.
Whenever a change in content is detected, our extension forwards the complete content to our OS-level service and into the prompt menu's preview field.
Once no changes have been detected for a predefined duration, it will signal to our background service that the response is complete.
We opted for this procedure, because chat interfaces perform various operations on the returned raw response throughout, such as replacing symbols.



  §.§.§ LLM chat interfaces and changes


has default tag names and element paths configured for the currently supported chat interfaces of ChatGPT, Gemini, and Mistral.
In the case that a chat interface changes, our extension  first uses heuristics to rediscover the elements (looking for the , , and an overlap in  added, or the same path in the DOM tree.
If this fails, our extension reports a problem and provide a tool to select the input box and the output element manually (similar to Chrome's `Select an element to inspect' function) to update our settings.
Because all parameters and URLs are stored in the extension settings, users could also add further LLM chat interfaces in the future.



  §.§.§ Browser handling to optimize for responsiveness


When a user triggers 's prompt menu, our browser extension opens the LLM's web page in a browser background window.
If the user has interacted with the current foreground app with this title before using , our browser extension loads the corresponding previous chat, such that the user can continue prompting within the context of the previous interaction sequence.
This accounts for loading times while the user still sees and operates 's prompt menu, ensuring that upon submitting a query our tool does not add to the response latency.



  §.§.§ Prompt padding


For submitting the prompt entered in 's prompt menu (or that associated with a menu item), we augment the text selection as follows:

Your task is to answer the following query from the user. Do not express approval or your own judgment of the query. Just respond with a clear answer. If prompted for code, just output the code, no explanation, just one response of code and nothing else.

User query: 

The user's query refers to this specific text:


The user issued the query while working with 

The user has provided additional context for their query. Do not directly quote this context, but use it to formulate a response.

Context: 





 §.§ Direct LLM interface via API (optional)


As an alternative to browser-based LLM access, also supports direct interaction with LLM backends via their APIs.
While this incurs query-specific charges for each use of , it affords more freedom over query types while still bringing LLM support to many apps.
Responses are also faster than those from the browser-based interface without variable-speed delays.



§ USER STUDY


The purpose of our study was to evaluate the efficacy of in the context of authoring, reading, and coding tasks.
In a controlled user study, we compared participants' performance solving these tasks using one of two Interfaces, and (version 3.5 through the web interface).




 §.§ Tasks


Participants completed three tasks during the study.
Each task had two instantiations that were similar in difficulty and format but varied in specific content.




  §.§.§ Writing


The writing task comprised three common subtasks for intelligent writing assistants: summarizing, editing, and text composing <cit.>.
For summarizing, participants summarized a paragraph from an academic paper draft in Overleaf.
For editing, they rewrote a narrative paragraph in Microsoft Word in a different style.
For composing, they wrote an email in Microsoft Outlook given a use-case: an invitation to an after-work celebration and an out-of-office announcement for a personal vacation.




  §.§.§ Reading


For the purpose of answering a question, participants read a one page long folk story in a foreign language that they were not familiar with in Acrobat Reader.
Their task was to answer a question about the text in English.



  §.§.§ Coding


Participants completed a simple coding task using Python in VSCode similar to previous studies <cit.>.
The task involved extracting information from a CSV file, aggregating dataframes, and writing to a text file as output.





 §.§ Procedure


The study started with a general introduction to the study context and explained the tasks to complete.
Participants received an introduction to and demonstration of and , and they were allowed to train in both for up to 20 minutes, using any native or web apps they wanted.

Participants then moved on to the main part of the study, for which they received written instructions.
Participants completed all three tasks using the same condition, took a break, and repeated the tasks in the second instantiation with the other condition.
The conditions and were counterbalanced across participants.
During each (sub)task, a timer on the screen displayed the remaining time to participants.

After completing all tasks using one interface, participants filled out a questionnaire that included SUS <cit.> and NASA TLX <cit.>.
When participants had finished both conditions, they had the option to provide qualitative feedback about both conditions.

The study ran on a Windows 10 computer with a full keyboard, mouse, and a 27 display.
An experimenter monitored participants' performance, answered questions, and ensured correct task outcomes.
All participants completed the study in less than an hour.




 §.§ Participants


We recruited 14 participants from various departments within our institution (5 female, 9 male, ages 22–37 years).
All participants had prior experience with Python. 
More than half of them reported that they use ChatGPT and other LLM-based tools (e.g., Copilot) multiple times a day. 
Coding is the most popular reported task for LLM use, and 12 participants that they routinely use ChatGPT for coding.
9 mentioned using it for writing, editing, and conducting research.
7 reported using ChatGPT for drafting emails and messages. 
Participants received a small gratuity for their time after the study.





 §.§ Measurements


We recorded the following metrics to assess participants' performance and impressions during the study:
(sub)task completion time, time-stamped logs during task completion, Likert-scale questionnaire responses, and qualitative comments.




  §.§.§ Completion time


The timer started once participants had completed reading the instructions of each task and had asked questions if they had any.
The timer stopped only when the experimenter had verified the correct completion of the task.
For writing, the text needed to be complete with no placeholders remaining (as often includes them).
For reading and question answering, the answer needed to be correct, otherwise participants could try again.
For coding, their code needed to produce the correct output, for which we provided a verification script that they ran on the command line.
We measure all completion times in seconds.



  §.§.§ System Usability Scale


We assessed the usability of and for daily desktop-based tasks through the system usability scale (SUS) <cit.>.
In particular, we scored the outcome of the SUS using the following standard formula <cit.>:
2.5 ∗ ( 20 + ∑_i ∈ 1,3,5,7,9SUS_i  - ∑_i ∈ 2,4,6,8,10SUS_i ),
where SUS_i denotes the i^th question in the SUS for i∈ 1, …,10.




  §.§.§ Task Load Index


We used the NASA task load index (NASA TLX) questionnaire <cit.> to assess participants' perceived workload while using LLM interfaces.
It consists on six sub-scales on mental demand, physical demand, temporal demand, performance, effort, and frustration level.
To increase the convenience for study participants, we adapted each sub-scale to a 5-point Likert item ranging from `very low' to `very high'.
We then calculated the load index via the following formula, which mimics the standard aggregation of the NASA TLX <cit.>: TLX = 25/6∑_i∈ 1, …, 6(TLX_i -1), where TLX_i is the i^th sub-scale of our adapted NASA TLX questionnaire.



  §.§.§ Questionnaire


In addition to SUS and NASA TLX, participants provided their experience in using LLM-based support for the tasks (following 's study <cit.>). 
A 5-point Likert scale assessed the condition's perceived impact on their efficiency and productivity, how often they needed to edit the output content to meet task requirements, and the follow-up work needed to reformat the output to produce the final result.



  §.§.§ Feedback


After the study, participants could report any feedback they had on either interface.
Participants commented on the task design, UI design, and the experience of solving the tasks using and . 
After their initial comments, we followed up with questions as reported in previous work <cit.>.



 §.§ Results




  §.§.§ Completion time


Figure <ref> shows all completion times.
We could not find a statistically significant difference of total completion time across all tasks (: M=382.21, SD=127.09; : M=436.07, SD=167.19; 80.5=0.43).
We did find a main effect of Interface on Completion Time during the editing task (-2.18<0.05), where participants completed the task significantly faster using (M=31.71, SD=19.04) than when using (M=51.14, SD=32.50). 
We did not observe any other significant differences.



  §.§.§ System Usability Scale


We found a significant effect of Interface on System Usability across all tasks (2.64<0.01).
Post-hoc tests showed that participants rated (M=62.54, SD=9.2) significantly higher than (M=51.68, SD=10.94) (<ref>).



  §.§.§ Task Load Index


We did not find a statistically significant overall difference for NASA TLX (-1.03=0.30).
Participants on average scored with 22.32 (SD=14.58) and with 17.26 (SD=11.77).

However, participants rated significantly lower on the effort-related sub-scale of the NASA TLX for ease of use (M=1.64, SD=0.49) compared to (M=2.27, SD=1.03).
We found no other significant difference between sub-scales of the NASA TLX.




  §.§.§ Questionnaire


We could not find a statistically significant difference in the questionnaire answers.
Participants' reported answers for efficiency were : M=4.64 (SD=0.63) vs. : M=4.50 (SD=0.65).
For perceived quality of their work, they rated with M=4.43 (SD=0.65) and with M=4.21 (SD=0.89).
On perceived difficulty, they rated with M=1.71 (SD=0.61) compared to (M=1.57, SD=0.65).
For number of needed modifications they needed to perform, they rated with M=2.43 (SD=0.51) and with  M=3.07 (SD=1.21) and for “less follow-up work was required” they scored : M=2.21 (SD=0.80) vs. : M=2.64 (SD=1.28).




  §.§.§ Feedback


Participants expressed that they felt more efficient when conducting the tasks with , because it eliminates the need for context switching. 
P1 said “Because is more integrated into the environment, I felt less the need to switch my focus compared to when I use .” 
The shortcuts for menu initiation and insertion were also appreciated. 
For example, P7 said “It is cool that no copy paste is required when using the tool.”

However, some participants still prefer the interface for user-friendliness. 
P6 pointed out the personalization of saying “talks in a more friendly manner and feels more like a personal assistant.”. 
P2 found the code generated by the web app easier to read.





§ DISCUSSION


Overall, the results of our evaluation were encouraging for our approach.
While achieved a significant difference in performance only for the editing task, interestingly with a 40% faster completion time on average, participants' ratings on the SUS and for ease-of-use on TLX confirmed our original motivation.
It is worth pointing out that all participants were at least well familiar with and that the majority of them used it frequently, such that we can assume that they were trained to the point of routine operation.
Therefore, was a challenging baseline despite the training participants received for .


One observation during the study was that participants' use of varied based on personal preference.
Some heavily relied on it, copying & pasting in large amounts of contents, while others use them more deliberately and sparingly.
For example, when replacing the placeholders in the composed emails, more than half of the participants insisted and requested in a prompt that this be done by the LLM, while the minority just filled them out manually. 


During or after the study, no participant commented on the choice of shortcuts or other design choices of , except for the 2 who commented on the lack of a `personalized assistant experience'.
Taken together, we interpret this as an encouraging sign that our design objectives for the UI and the interaction did not interfere with participants' workflows.
On the contrary, participants offered several recommendations how the UI could be extended to better support the tasks at hand.
A frequent suggestion was that the menu buttons (1..5) should be generated based on the context of the selected text and application window.







§ LIMITATIONS AND FUTURE WORK




  §.§.§ In-situ editing


Participants took time to get used to 's UI, as LLM responses flowed directly into the foreground app upon pressing .
Despite the preview in our UI, they were used to seeing the formatted output in the context of previous responses and using copy & paste to transfer (portions of) the response.
Future implementations could include collapsible displays of history for previously generated responses, syntax highlighting, or the option to insert only portions of the previewed response.




  §.§.§ vs. apps with built-in LLM support


While our approach provides general-purpose text generation and editing across apps, those with built-in LLM assistance can offer better tailored support.
For example, Github Copilot's Codex is specifically integrated into programming environments (e.g., VSCode, JetBrains).
Similarly, assistants in creativity apps can generate and manipulate app-specific objects (e.g., Adobe Illustrator or Premiere), where mere text responses are insufficient.



  §.§.§ Context is limited to text


Our current implementation is limited to sourcing text from the surrounding context.
Future work could additionally include screenshots to GPT-4, such as to capture a screenshot of the rendered draft when editing in Overleaf or capturing surrounding images in InDesign for suggestions.




  §.§.§ Integration with other web-based query services


's focus is on LLM-based chat interactions, but our implementation can also interface with other web-based services, such as Google Translate and DeepL (see video), as query elements and response fields are equally well defined on the web page.
Future implementations could explore more general media types (e.g., Wolfram Alpha).



§ CONCLUSION


We have presented , a system-wide bridge between applications and LLM-based assistance for textual content.
By pressing a shortcut, our system shows a lightweight overlay to query LLM chat systems with a text selection and allows the response to seamlessly transfer back into the app.
In the background, our system interfaces with browser-based LLM chat interfaces to submit the query and process the response or it can query LLM backends directly through their APIs.
thus allows users to leverage LLM capabilities for text manipulation across virtually all apps, even if an app does not build in native LLM support.

In our comparison of with ChatGPT, we found statistically significant differences only in its ability to support text editing more quickly as part of a rewrite task and its perceived usability (rated on SUS <cit.>).
Given these results and the qualitative insights during the study as the basis for future developments, we believe that is a promising option for supporting text manipulation tasks and supporting quality of work across applications, without investing in app-specific LLM subscriptions.


_templates/ACM-Reference-Format




 Paper Title: FedBChain: A Blockchain-enabled Federated Learning Framework for Improving DeepConvLSTM with Comparative Strategy Insights.



High-order quasi-interpolation with generalized Gaussian kernels restricted over tori
label1
    Changwei Wanglabel2
    August 5, 2024
============================================================================================

empty
empty



§ ABSTRACT 

Recent research in the field of Human Activity Recognition has shown that an improvement in prediction performance can be achieved by reducing the number of LSTM layers. However, this kind of enhancement is only significant on monolithic architectures, and when it runs on large-scale distributed training, data security and privacy issues will be reconsidered, and its prediction performance is unknown. In this paper, we introduce a novel framework: FedBChain, which integrates the federated learning paradigm based on a modified DeepConvLSTM architecture with a single LSTM layer. This framework performs comparative tests of prediction performance on three different real-world datasets based on three different hidden layer units (128, 256, and 512) combined with five different federated learning strategies, respectively. The results show that our architecture has significant improvements in Precision, Recall and F1-score compared to the centralized training approach on all datasets with all hidden layer units for all strategies: FedAvg strategy improves on average by 4.54%, FedProx improves on average by 4.57%, FedTrimmedAvg improves on average by 4.35%, Krum improves by 4.18% on average, and FedAvgM improves by 4.46% on average. Based on our results, it can be seen that FedBChain not only improves in performance, but also guarantees the security and privacy of user data compared to centralized training methods during the training process. The code for our experiments is publicly available (https://github.com/Glen909/FedBChain).





§ INTRODUCTION


Human activity recognition(HAR) [1] has a wide range of applications and importance in several fields. In the field of digital health, recognizing human activities, especially for the elderly, can help maintain or improve their health; in the field of smart home, HAR can help automation systems better coordinate collaboration among devices in the home; in the field of human-computer interaction, clearly recognizing human activities can better understand human intentions and make timely and accurate responses. In a world where machine learning and deep learning are becoming increasingly fast and iterative, a larger number of researchers are applying them to the field of HAR to empower the recognition function and make it more efficient.

Among these researchers, DeepConvLSTM proposed by [2] is one of the deep learning architectures that has attracted a lot of attention in the study of HAR. This innovative architecture incorporates convolutional and recurrent layers. It shows excellent performance and outstanding results on several real-world datasets (Opportunity [3] and Skoda Mini Checkpoint datasets [4]). Specifically, [2] designed the LSTM layers in the DeepConvLSTM architecture as a two-layer structure and 128 more hidden units in each layer. As pointed out by [5] and further confirmed by [6]. It is generally accepted that at least two LSTM layers are required for optimal modelling during sequential data processing in the field of HAR.

However, this was doubted and reassessed by [7], whose study used a single LSTM layer in a recurrent layer structure for model training. The results showed that the DeepConvLSTM structure based on a two-layer LSTM was outperformed by the single-layer DeepConvLSTM (with the LSTM module of the second layer removed) in terms of accuracy, recall, and F1-scores for four of the five datasets. Specifically, [7] achieved an average of 62% fewer trainable parameters across the architecture with this adaptation and reduced the overall training time by 48%. Although [7] achieved remarkable success on the DeepConvLSTM architecture based on single-layer LSTM, it is only applicable to monolithic servers, and in the large-scale distributed training environments, even if the performance of monolithic architectures is optimized again, it cannot effectively stifle the security and privacy issues of personal data. Therefore, in order to better handle the privacy and security of user data during training, federated learning [8] is proposed as an effective solution in large-scale distributed training environments. Federated learning is a decentralized machine learning technique that allows collaborative model training among multiple data owners while the data is always stored locally at the data owner.

Therefore, this project develops FedBChain, a federated learning framework specialized for large-scale distributed training environments, on top of DeepConvLSTM. This framework employs distributed training instead of the centralized training approach, thus enhancing data privacy and security. In addition, compared to the general federated learning process, FedBChain employs blockchain technology [9] as the global model in designing the global model. Generally, federated learning usually relies on a central node for aggregation and distribution of model parameters, which may have a single point of failure that jeopardizes the overall system performance, therefore, the design of this framework uses blockchain technology instead of a centralized global model. Meanwhile, the deployment and execution of smart contracts automatically performs model aggregation and distribution of the latest parameters to further ensure the robustness and availability of the system.

The contributions of this paper are summarized as follows:



  * This project introduces the FedBChain framework, which is based on a single-layer DeepConvLSTM architecture for federated learning training approach to ensure that data security and privacy are enhanced throughout the training process. The framework is suitable for large-scale distributed federated training.

  * Within the FedBChain framework, this project integrates a blockchain module to replace the centralized global model. This modification minimizes the risk of single point of failure and ensures that each model parameter update is securely recorded on the blockchain, making it traceable, reproducible and tamper-proof.

  * Based on the proposed FedBChain framework, this project analyses the DeepConvLSTM under five different federated learning strategies in comparison with centralized training methods, and tests the DeepConvLSTM on three datasets with three different configurations of hidden layer units. The results show that FedBChain significantly outperforms the centralized training method.




§ RELATED WORK




 §.§ Federated Learning


Federated Learning (FL) has risen as a crucial framework for decentralized machine learning, designed to address data privacy issues while leveraging data from diverse sources for model training. Initially introduced by [10] in 2016, this approach ensures that data remains on local devices, with only model updates being shared to a central server, thereby significantly diminishing the risks associated with centralized data storage. Subsequent research has broadened to tackle various challenges in FL, including minimizing communication overhead as discussed by [11], ensuring equitable resource distribution among participants as explored by [12], and fortifying model security against adversarial attacks in distributed environments as identified by [13]. Additionally, advancements have been made with techniques that employ sophisticated encryption methods like secure multi-party computation and differential privacy to enhance the privacy protections of federated models, as demonstrated by [14].



 §.§ Human Activity Recognition


HAR has garnered significant attention due to its applications in health monitoring and automated systems. Traditionally, HAR systems have relied heavily on sensor data, which is aggregated and processed on centralized servers [15]. However, privacy concerns and the logistical complexities associated with centralized data aggregation have spurred the exploration of decentralized methodologies, particularly through federated learning. For example, [16] illustrated the application of federated learning in HAR using mobile sensors, substantially enhancing privacy by keeping sensitive data on the devices themselves. More recent studies have furthered the field by integrating deep learning techniques to boost recognition accuracy while maintaining the decentralized framework of federated learning [17]. These developments suggest a shift toward more secure and efficient methods for implementing HAR in privacy-sensitive environments.



 §.§ Blockchain


In the realm of HAR, the integration of blockchain technology has surfaced as an innovative trend aimed at bolstering system security and privacy. For example, [18] showcased a blockchain-based HAR system that employs smart contracts to verify and log sensor data, thereby enhancing transparency and trust in data management. [19] investigated how blockchain could safeguard user privacy through its distributed ledger technology, mitigating risks linked to centralized data storage. Meanwhile, [20] merged machine learning with blockchain to strengthen the immutability and automation of verification processes for HAR data on smart devices. Collectively, these studies not only underscore blockchain's potential to fortify privacy and security but also illustrate its role in improving the efficiency and transparency of data management.



§ METHODS




 §.§ Datasets
 To evaluate the effectiveness of the DeepConvLSTM architecture after integrating the federated learning strategy, this project uses three well-known datasets processed in the same way as [7]: the Heterogeneous Activity Recognition (HHAR) dataset [21], the Real-World HAR (RWHAR)[22], and the Smartphone-based Human Activity and Pose Transition Recognition ( SBHAR) [23].

RWHAR: The RWHAR dataset contains data from 15 participants performing eight different activities (going up stairs, going down stairs, jumping, lying down, standing, sitting, running, and walking), as well as an empty class. According to the experimental parameters mentioned in [7], the present analysis was limited to 3D acceleration data recorded at 50Hz using wrist-worn sensors.

SBHAR: The SBHAR dataset contains data from 30 individuals engaged in daily activities such as standing, sitting, lying down, walking, walking up stairs and walking down stairs as shown in [23]. In addition to these six activities, the dataset includes six types of postural transitions (from standing to sitting, from sitting to standing, from sitting to lying, from lying to sitting, from standing to lying, and from lying to standing) and an empty category. To maintain comparability, this analysis also focuses only on the raw 3D acceleration data collected at a 50Hz sampling rate.

HHAR: Similar to the RWHAR dataset, the HHAR dataset contains data from nine individuals involved in daily activities, including six activities (cycling, sitting, standing, walking, and climbing up and down stairs) and the null category for prediction purposes. Consistent with the methodology of the previous dataset, the current analysis focused solely on 3D acceleration data recorded at 100Hz by a wrist-worn sensor unit.

   



 §.§ Architecture

The novel framework, FedBChain, is illustrated in Fig. 1 and is structured around M servers and N clients. The M servers collaboratively establish a global model that is tasked with gathering model parameters, which are locally trained by the clients. Following specific strategies, these parameters are aggregated, and the resultant model parameters are subsequently redistributed to each client. The local models at the client sites then update their parameters from the global model before continuing training on local data. This cycle repeats until the model reaches convergence. Throughout this process, local data of the clients remains exclusively local and only weight parameters of the model are transmitted to the global model for aggregation.

Specifically, the FedBChain leverages the Flower framework [24] for implementation, where model parameter aggregation tasks are managed using FedAvg [25], FedProx [26], FedTrimmedAvg [27], Krum [28], and FedAvgM [29] strategies. Additionally, we evaluate these strategies across three different datasets and three distinct hidden layer sizes (128, 256, and 512) as depicted in Fig. 2. The baseline model for this experiment is established using the centralized training method based on the DeepConvLSTM architecture, with equivalent hidden layer units applied across the datasets.

To ensure consistent result comparability, in line with [7], this project uses an Adam optimizer with a reduced weight decay (1e-6) and learning rate (1e-4), and employ Glorot initialization [30] for network weight setup. During the training, Five-Fold Cross-Validation was applied to calculate loss, enabling the network to adapt to unbalanced datasets. This study primarily investigates differences between centralized and federated training methods and compares outcomes across various federated strategies without excessively adjusting the hyperparameters for the RWHAR, SBHAR, and HHAR datasets. However, since the HHAR dataset samples at 100 Hz, we adjust the size of the convolution filters to 21 to preserve the relationship between filter size and the sliding window duration, ensuring that each filter captures a consistent amount of information across all datasets.

In this project, Hyperledger Fabric [31] will be deployed to structure the decentralized global model consisting of M servers. This decentralized network configuration will include two peer nodes, one order node, and one channel, namely Ledger. The process unfolds as follows: Initially, the global model parameters are proposed to the peer nodes as the inaugural proposal, where they are processed using the Chain Code installed on these nodes. The order node then encapsulates this proposal into the genesis block, which is subsequently dispatched to each peer node for independent verification. Once verified, the proposal is added to the ledger. This procedure is replicated for each round of global model updates, thereby ensuring the traceability, trackability, and security of the model parameters. Additionally, this approach mitigates the risk of single points of failure, enhancing overall system stability.

   



§ RESULTS





In both centralized and federated learning training methods, K-Fold Cross-Validation (specifically Five-Fold) is uniformly applied across each dataset. This approach ensures that every data point serves both as part of the training and the validation set, which is particularly advantageous when data volumes are limited. Additionally, the data preprocessing in federated training is consistent with that of centralized methods across each dataset.

As illustrated in Fig. 2, this study compares the training results of three different hidden layer units (128, 256, and 512) using a single-layer DeepConvLSTM architecture under centralized and federated learning methods (across five different strategies). The results for each strategy represent the average performance of three clients, with outcomes averaged over five runs. This experiment employs precision, recall, and F1-score as the evaluation metric.

Table I displays that all five federated learning strategies outperformed the centralized training approach on HHAR. Notably, FedAvgM achieved superior performance not only compared to centralized training but also excelled over the other strategies under various hidden unit settings. Moreover, with 128 hidden units, FedProx delivered Precision, Recall, and F1 scores of 80.78%, 81.10%, and 80.81% respectively, showing a significant improvement over the other three federated strategies. With 256 hidden units, FedAvg surpassed the others, with increases of 5.84%, 5.85%, and 5.90% in the three metrics respectively. Therefore, when implementing DeepConvLSTM on a single layer using one of these five federated learning strategies, the overall performance on the HHAR dataset exceeded that of the centralized approach, with specific outcomes depending on the number of hidden units.

In the RWHAR training set(Table II), the outcomes of the five federated training methods consistently surpass those of centralized training. Unlike the results seen in the HHAR dataset, for the RWHAR dataset, irrespective of the federated strategy employed, the F1-score for models with 128 hidden layer units consistently outperforms those with 256 and 512 units. For instance, under federation strategy FedAvg, the F1-score improvement at 128 units is 4.49%, compared to 4.01% and 4.18% for 256 and 512 units, respectively. Similarly, in FedAvgM, the F1-score for 128 units rose by 4.56%, while increases for 256 and 512 units were 4.01% and 4.24%, respectively.







Contrasting with these two datasets, the SBHAR dataset(Table III) does not show a significant improvement in recall. Notably, when employing 512 hidden layer units under Krum, there is even a slight decline of 0.02%. Specifically, in FedAvg, the recall for 128 units increased by 3.18%, for 256 by 1.16%, and for 512 by 1.87%. In FedProx, the recall enhancements for 128 and 256 hidden units are more pronounced, at 4.2% and 2.45% respectively, while the improvement for 512 units is only 0.54%. Therefore, in the SBHAR dataset, reducing the number of hidden layers significantly enhances performance improvements. This trend is also evident across different federated strategies.




Thus far, we perform a side-by-side comparison of the experimental results, i.e., a comparison of the results of the five federated learning strategies on Precision, Recall, and F1-score based on three different hidden layer units running on three datasets. The results(Table IV) show that FedAvg improves the most significantly on Precision, i.e., 4.75%. Compared to the other strategies, FedTrimmedAvg performs best on Recall, with an improvement of 0.55 percentage points compared to Krum. And on F1-score, the best strategy for improvement was FedProx, which improved by 4.85% compared to the centralized training method.

In summary, regardless of the federated learning strategy adopted, results consistently outperform centralized training methods. Specifically, on the HHAR dataset with 512 hidden layer units, the five federated learning strategies yielded average improvements of 5.51% in precision, 5.52% in recall, and 5.57% in F1-score. On the RWHAR dataset with 128 hidden layer units, the strategies averaged improvements of 4.25% in precision, 4.60% in recall, and 4.46% in F1-score. Meanwhile, on the SBHAR dataset with 256 hidden layer units, the improvements were 4.76% in precision, 2.24% in recall, and 5.12% in F1-score. These results highlight nuanced variations across different federated learning strategies and datasets, influenced by the number of hidden layer units employed.



§ CONCLUSION AND FUTURE WORK

In conclusion, this paper explores federated transformation and centralized training using the single-layer LSTM architecture proposed by [12], applying the Flower framework to execute five distinct federated learning strategies across three real datasets, each with three varying sizes of hidden units (128, 256, and 512). The experimental results demonstrate that, compared to centralized training methods, the single-layer DeepConvLSTM employing federated learning exhibits improvements in Precision, Recall, and F1-Score to varying extents. Additionally, this project also conducted a horizontal comparison of performance enhancements across different federated strategies, FedAvg, FedTrimmedAvg and FedProx have different levels of improvement in Precision, Recall and F1-score evaluation criteria respectively compared to other strategies.

Furthermore, This paper has replaced the centralized global model in federated learning with a decentralized structure using Hyperledger Fabric. This adjustment not only mitigates the risk of single points of failure but also enhances the tamper-proofing and traceability of model parameters.

In ongoing research, we aim to further bolster security and privacy in the federated learning process. On one hand, we plan to employ advanced encryption protocols, such as homomorphic encryption or secure multi-party computation, during data transmission between clients and the server to protect against theft and reverse engineering of model parameters. On the other hand, we seek to refine the consensus strategy within the server's decentralized structure to enhance the verification efficiency and processing speed of the entire network. Through these improvements, we hope to effectively advance the application of federated learning and blockchain technologies in the wearable sector, significantly enhancing the protection of user privacy and security.



99

c1 G. Saleem, U. I. Bajwa, and R. H. Raza, “Toward human activity recognition: a survey,” Neural Computing and Applications, Oct. 2022, doi: https://doi.org/10.1007/s00521-022-07937-4.
c2 F. Ordóñez and D. Roggen, “Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition,” Sensors, vol. 16, no. 1, p. 115, Jan. 2016, doi: https://doi.org/10.3390/s16010115.
c3 D. Roggen et al., “Collecting complex activity datasets in highly rich networked sensor environments,” 2010 Seventh International Conference on Networked Sensing Systems (INSS), Jun. 2010, doi: https://doi.org/10.1109/inss.2010.5573462.
c4 P. Zappi et al., “Activity Recognition from On-Body Sensors: Accuracy-Power Trade-Off by Dynamic Sensor Selection,” Lecture Notes in Computer Science, pp. 17–33, doi: https://doi.org/10.1007/978-3-540-77690-1_2.
c5 K. Chen, D. Zhang, L. Yao, B. Guo, Z. Yu, and Y. Liu, “Deep Learning for Sensor-based Human Activity Recognition,” ACM Computing Surveys, vol. 54, no. 4, pp. 1–40, May 2021, doi: https://doi.org/10.1145/3447744.
c6 Andrej Karpathy, J. C. Johnson, and L. Fei-Fei, “Visualizing and Understanding Recurrent Networks,” arXiv (Cornell University), Jun. 2015, doi: https://doi.org/10.48550/arxiv.1506.02078.
c7 M. Bock, A. Hoelzemann, M. Moeller, and K. Van Laerhoven, “Improving Deep Learning for HAR with shallow LSTMs,” arXiv.org, Sep. 21, 2021. https://arxiv.org/abs/2108.00702.
c8 Y. Liu et al., “Vertical Federated Learning: Concepts, Advances, and Challenges,” IEEE transactions on knowledge and data engineering, pp. 1–20, Jan. 2024, doi: https://doi.org/10.1109/tkde.2024.3352628.
c9 A. Qammar, A. Karim, H. Ning, and J. Ding, “Securing federated learning with blockchain: a systematic literature review,” Artificial Intelligence Review, Sep. 2022, doi: https://doi.org/10.1007/s10462-022-10271-9.
c10 B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Aguera y Arcas, "Communication-Efficient Learning of Deep Networks from Decentralized Data," in Proc. 20th Int. Conf. Artif. Intell. Statist. (AISTATS), PMLR 54, 2017, pp. 1273-1282.
c11 J. Konečný, H. Mcmahan, F. Yu, A. Theertha, D. Google, and P. Richtárik, “FEDERATED LEARNING: STRATEGIES FOR IMPROVING COMMUNICATION EFFICIENCY.” Available: https://arxiv.org/pdf/1610.05492.pdf.
c12 T. Li, M. Sanjabi, A. Beirami, and V. Smith, “Fair Resource Allocation in Federated Learning,” arXiv:1905.10497 [cs, stat], Feb. 2020, Available: https://arxiv.org/abs/1905.10497.
c13 K. Bonawitz et al., “Practical Secure Aggregation for Privacy-Preserving Machine Learning,” Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Oct. 2017, doi: https://doi.org/10.1145/3133956.3133982.
c14 R. C. Geyer, T. Klein, and M. Nabi, “Differentially Private Federated Learning: A Client Level Perspective,” arXiv:1712.07557 [cs, stat], Mar. 2018, Available: https://arxiv.org/abs/1712.07557.
c15 A. Bulling, U. Blanke, and B. Schiele, “A tutorial on human activity recognition using body-worn inertial sensors,” ACM Computing Surveys, vol. 46, no. 3, pp. 1–33, Jan. 2014, doi: https://doi.org/10.1145/2499621.
c16 G. Gad and Z. Fadlullah, “Federated Learning via Augmented Knowledge Distillation for Heterogenous Deep Human Activity Recognition Systems,” Sensors, vol. 23, no. 1, p. 6, Dec. 2022, doi: https://doi.org/10.3390/s23010006.
c17 Nikola Simić et al., “Enhancing Emotion Recognition through Federated Learning: A Multimodal Approach with Convolutional Neural Networks,” Applied sciences, vol. 14, no. 4, pp. 1325–1325, Feb. 2024, doi: https://doi.org/10.3390/app14041325.
c18 A. Khatoon, “A Blockchain-Based Smart Contract System for Healthcare Management,” Electronics, vol. 9, no. 1, p. 94, Jan. 2020, doi: https://doi.org/10.3390/electronics9010094. 
c19 Q. Liu, Y. Liu, M. Luo, D. He, H. Wang, and K.-K. R. Choo, “The Security of Blockchain-Based Medical Systems: Research Challenges and Opportunities,” IEEE Systems Journal, pp. 1–12, 2022, doi: https://doi.org/10.1109/JSYST.2022.3155156.
c20 D. M. Patel, C. K. Sahu, and R. Rai, “Security in modern manufacturing systems: integrating blockchain in artificial intelligence-assisted manufacturing,” International Journal of Production Research, pp. 1–31, Sep. 2023, doi: https://doi.org/10.1080/00207543.2023.2262050. 
c21 A. Stisen et al., “Smart Devices are Different,” Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, Nov. 2015, doi: https://doi.org/10.1145/2809695.2809718.
c22 Timo Sztyler and Heiner Stuckenschmidt, “On-body localization of wearable devices: An investigation of position-aware activity recognition,” Mar. 2016, doi: https://doi.org/10.1109/percom.2016.7456521.
c23 M. Núñez-Regueiro, “Yaşlı Kadınlarda Üreme Sağlığı,” DergiPark (Istanbul University), vol. 1, no. 1, Feb. 2015, doi: https://doi.org/10.1016/j.
c24 D. J. Beutel et al., “Flower: A Friendly Federated Learning Research Framework,” arXiv:2007.14390 [cs, stat], Apr. 2021, Available: https://arxiv.org/abs/2007.14390.
c25 M. H. Brendan, E. Moore, D. Ramage, S. Hampson, and Arcas, Blaise Agüera y, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” arXiv.org, 2016. https://arxiv.org/abs/1602.05629.
c26 T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated Optimization in Heterogeneous Networks,” arXiv:1812.06127 [cs, stat], Apr. 2020, Available: https://arxiv.org/abs/1812.06127.
c27 D. Yin, Y. Chen, K. Ramchandran, and P. Bartlett, “Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates,” arXiv.org, Feb. 25, 2021. https://arxiv.org/abs/1803.01498.
c28 P. Blanchard, E. M. E. Mhamdi, R. Guerraoui, and J. Stainer, “Byzantine-Tolerant Machine Learning,” arXiv.org, Mar. 08, 2017. https://arxiv.org/abs/1703.02757.
c29 T.-M. H. Hsu, H. Qi, and M. Brown, “Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification,” arXiv:1909.06335 [cs, stat], Sep. 2019, Available: https://arxiv.org/abs/1909.06335.
c30 X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” proceedings.mlr.press, Mar. 31, 2010. http://proceedings.mlr.press/v9/glorot10a.
c31 E. Androulaki et al., “Hyperledger fabric,” Proceedings of the Thirteenth EuroSys Conference on - EuroSys ’18, 2018, doi: https://doi.org/10.1145/3190508.3190538.



 Paper Title: Human-Data Interaction Framework: A Comprehensive Model for a Future Driven by Data and Humans.


Uplink Wave-Domain Combiner for Stacked Intelligent Metasurfaces Accounting for Hardware Limitations
    Maryam Rezvani1, Raviraj Adve1, Akram bin Sediq2, Amr El-Keyi2
	1Dept. of Electrical and Computer Engineering,
		University of Toronto, 
		Toronto, Canada 
	
	2Ericsson Canada, 
		Ottawa, Canada, 

		emails: {mrezvani, rsadve}@ece.utoronto.ca, {akram.bin.sediq, amr.el-keyi}@ericsson.com

    August 5, 2024
====================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
	In an age defined by rapid data expansion, the connection between individuals and their digital footprints has become more intricate. The Human-Data Interaction (HDI) framework has become an essential approach to tackling the challenges and ethical issues associated with data governance and utilization in the modern digital world. This article outlines the fundamental steps required for organizations to seamlessly integrate HDI principles, emphasizing auditing, aligning, formulating considerations, and the need for continuous monitoring and adaptation. Through a thorough audit, organizations can critically assess their current data management practices, trace the data lifecycle from collection to disposal, and evaluate the effectiveness of existing policies, security protocols, and user interfaces. The next step involves aligning these practices with the main HDI principles, such as informed consent, data transparency, user control, algorithm transparency, and ethical data use, to identify gaps that need strategic action. Formulating preliminary considerations includes developing policies and technical solutions to close identified gaps, ensuring that these practices not only meet legal standards, but also promote fairness and accountability in data interactions. The final step, monitoring and adaptation, highlights the need for setting up continuous evaluation mechanisms and being responsive to technological, regulatory, and societal developments, ensuring HDI practices stay up-to-date and effective. Successful implementation of the HDI framework requires multi-disciplinary collaboration, incorporating insights from technology, law, ethics, and user experience design. The article posits that this comprehensive approach is vital for building trust and legitimacy in digital environments, ultimately leading to more ethical, transparent, and user-centric data interactions.









§ EXECUTIVE SUMMARY

This article presents a comprehensive field of Human-Data Interaction (HDI), addressing the complex challenges of ethical data management in the digital age. The HDI framework aims to balance the exponential growth of data with the need to respect individual rights, promote transparency, and ensure ethical data practices.
Key components of the framework include:


    
  * Foundational Principles: Emphasizing human agency, transparency, fairness, accountability, privacy, and stakeholder engagement.
    
  * Structural Components: Outlining data governance frameworks, technological solutions, and human-centric practices necessary for effective HDI implementation.
    
  * Implementation Roadmap: Providing a phased approach for organizations to integrate HDI principles, including assessment, policy development, implementation, and continuous adaptation.
    
  * Ethical and Practical Considerations: Addressing challenges such as data complexity, bias mitigation, user empowerment, and regulatory compliance.
    
  * Training and Education: Stressing the importance of cultivating an HDI-centric culture through comprehensive training programs for employees and education initiatives for users.
    
  * Monitoring and Adaptation: Highlighting the need for continuous evaluation and responsiveness to technological, regulatory, and societal shifts.


The article argues that adopting HDI principles is not only a moral imperative but also essential for building trust and legitimacy in digital ecosystems. It emphasizes the need for a multidisciplinary approach, combining insights from technology, law, ethics, and user experience design.
Future directions for HDI research and implementation are discussed, including the development of adaptive ethical frameworks, technological innovations, and strategies for global application.

This work contributes to the growing field of ethical data management by providing a structured approach for organizations to navigate the complexities of data interactions in a manner that respects human values and promotes social well-being.



§ INTRODUCTION


In today's rapidly evolving digital landscape, data has become the lifeblood of our interconnected world. The exponential growth of digital technologies has led to an era in which vast amounts of information are generated, collected, and analyzed at unprecedented scales. From smartphones in our pockets to the expansive networks of smart city infrastructure, data flows endlessly, shaping our daily lives in ways both visible and unseen. This data-driven revolution has led to remarkable advancements in fields ranging from healthcare and scientific research to commerce and public services. Predictive analytics inform critical business decisions, machine learning algorithms power personalized experiences, and big data insights drive policy making at national and global levels. The potential for innovation and progress seems boundless <cit.>.

However, this data-centric paradigm is not without its challenges. As our reliance on data-driven systems intensifies, we find ourselves grappling with profound ethical, privacy, and security concerns. The Cambridge Analytica scandal laid bare the vulnerabilities of personal data and its potential for manipulation. Algorithmic bias has raised alarming questions about fairness and discrimination in automated decision-making processes. Meanwhile, data breaches continue to erode public trust, highlighting the precarious nature of digital information security.
In addition, the complexity and opacity of many data systems have created a widening gap between those who control these technologies and those affected by them <cit.>. Individuals often find themselves navigating a digital landscape where their personal information is commodified, their behaviors are predicted and influenced, and their futures are increasingly shaped by algorithms they neither understand nor control.

In this context, a critical juncture is presented. How do we harness the transformative power of data while safeguarding individual rights, fostering transparency, and ensuring ethical practices? How can we build data ecosystems that not only drive innovation, but also empower individuals and promote societal well-being?

These pressing questions underscore the need for a fundamental shift in how we approach data interaction and management. It is against this backdrop that the concept of Human-Data Interaction (HDI) emerges as a crucial paradigm, offering a path towards a more equitable, transparent, and human-centric data future. In response to these pressing challenges, the concept of HDI emerges as a transformative paradigm and represents a fundamental shift in how we approach the relationship between individuals and the data ecosystems that increasingly shape our world. At its core, HDI aims to rebalance the power dynamics in data-driven systems, placing human agency, understanding, and values at the center of data practices <cit.>.

HDI goes beyond traditional notions of data privacy and security, proposing a holistic framework that encompasses the entire lifecycle of data interactions. It recognizes that individuals are not just subjects of data collection, but active participants in a complex data ecosystem. Using three key principles: legibility, agency and negotiability, HDI seeks to empower individuals with the ability to understand, control and negotiate their data relationships <cit.>.

Legibility addresses the often opaque nature of data systems, advocating for transparency and comprehensibility in how data are collected, processed, and used. The agency emphasizes the importance of meaningful control that allows individuals to make informed decisions about their data and its implications. Negotiability acknowledges the dynamic nature of data relationships, promoting flexible and context-sensitive approaches to data rights and permissions <cit.>.

By integrating these principles, HDI offers a framework for developing more ethical, transparent, and user-centric data practices. It challenges organizations to rethink their data strategies, moving beyond compliance-driven approaches to embrace a more collaborative and responsible ethos. For policymakers, HDI provides a lens through which to craft more nuanced and effective data governance frameworks.

Importantly, HDI is not just a theoretical construct, but a call to action. It requires practical implementation in the technological, organizational, and societal domains. By adopting HDI principles, It is possible to construct data ecosystems that facilitate innovation and advancement while simultaneously upholding human dignity, fostering trust, and contributing to the collective well-being of society..

As we navigate the complexities of our data-driven future, HDI offers a beacon of hope a pathway to harness the power of data while upholding the values that make us human. This article explores the foundations, implications, and practical applications of HDI, charting a course toward a more equitable and empowering data landscape for all.

This article aims to provide a comprehensive exploration and analysis of the Human-Data Interaction (HDI) framework, with the following key objectives and contributions:


    
  * Conceptual Clarification:  It provides a comprehensive analysis of the fundamental principles of HDR, offering a clear and accessible explanation of its core tenets legibility, agency, and negotiability. This contributes to a more nuanced understanding of HDI within the broader context of data ethics and governance.
    
  * Framework Development: it presents an expanded HDI framework that builds upon existing literature, integrating insights from diverse fields such as computer science, ethics, law, and social sciences. This interdisciplinary approach results in a more robust and applicable model for HDI implementation.
    
  * Implementation Roadmap: A significant contribution of this article is the development of a practical, step-by-step roadmap for organizations to adopt and integrate HDI principles. This guide addresses the gap between theoretical understanding and practical application of HDI.
    
  * Ethical and Practical Considerations:The ethical implications and practical challenges associated with the implementation of the HDR are analysed in depth. By doing so, It contributes to the ongoing dialogue on responsible data practices and offers strategies to overcome potential obstacles.
    
  * Future Directions: It identifies emerging trends and areas for future research in HDR, thereby contributing to the evolution of the field and highlighting opportunities for further academic and practical exploration.
    
  * Interdisciplinary Synthesis: By drawing connections between HDI and related fields such as data ethics, privacy law, and user experience design, we contribute to a more holistic understanding of data governance in the digital age.
    
  * Policy Implications:The potential impact of the widespread adoption of HDR on data protection policy and regulation is discussed, offering insights that can inform the development of future policy.


Through these objectives and contributions, this article aims to advance both the theoretical understanding and practical application of Human-Data Interaction, providing valuable insights for researchers, practitioners, policymakers, and organizations navigating the complexities of ethical data management in the 21st century.

The timeliness and significance of this work cannot be overstated, as HDI has become increasingly crucial in our current technological and societal landscape for several interconnected reasons:


    
  * Data Ubiquity and Complexity: The pervasiveness of data collection and processing in everyday life has created what Zuboff (2015) terms "surveillance capitalism" <cit.>. As our digital footprints expand, HDI provides a framework for managing the complex relationships between individuals and their data. Andrejevic (2014) argues that the sheer volume and variety of data collected about individuals necessitates new approaches to data governance and interaction <cit.>.
    
  * Algorithmic Decision-Making: With the rise of machine learning and AI, algorithms increasingly influence critical aspects of our lives. Mittelstadt et al. (2016) highlight the ethical challenges posed by algorithmic decision-making, including issues of transparency, fairness, and accountability <cit.>. HDI principles, particularly legibility, are crucial for addressing these challenges and ensuring that individuals can understand and challenge decisions made about them.
    
  * Privacy Concerns: High-profile data breaches and misuse cases have eroded public trust in data-driven systems. Auxier et al. (2019) report that a majority of Americans feel they have little to no control over their personal data <cit.>. HDI's emphasis on agency and negotiability offers a pathway to restore trust and give individuals more control over their data.
    
  * Regulatory Landscape: The introduction of comprehensive data protection regulations like the GDPR in Europe and the CCPA in California reflects growing societal concerns about data privacy. Cate and Mayer-Schönberger (2013) argue that traditional notice and consent models are insufficient in the big data era <cit.>. HDI provides a more nuanced and flexible approach to data governance that aligns with these evolving regulatory requirements.
    
  * Digital Divide and Data Literacy: As data-driven services become more integral to daily life, there's a risk of exacerbating existing inequalities. Ragnedda and Muschert (2013) discuss how differential access to and understanding of digital technologies can reinforce social disparities <cit.>. HDI's focus on legibility and agency can help bridge this gap by making data systems more accessible and understandable to all.
    
  * Ethical AI and Responsible Innovation: As AI systems become more advanced, ensuring their ethical development and deployment is paramount. Floridi et al. (2018) propose an ethical framework for AI that aligns closely with HDI principles <cit.>. By centering human values and agency, HDI contributes to the responsible development of AI and other data-driven technologies.
    
  * Data as a Human Right: There's growing recognition of data protection and privacy as fundamental human rights. Banisar and Davies (1999) argue for the importance of information privacy in maintaining human dignity and autonomy <cit.>. HDI provides a practical framework for operationalizing these rights in the digital sphere.
    
  * Personalization vs. Privacy Paradox: Users often express privacy concerns while simultaneously engaging in privacy-compromising behaviors for perceived benefits. Acquisti et al. (2015) explore this "privacy paradox" <cit.>. HDI's negotiability principle offers a way to navigate this tension by allowing for more nuanced and context-dependent data sharing decisions.


As we delve into the complexities of our data-driven world, it becomes increasingly clear that a new paradigm is needed to address the ethical, social, and technical challenges that arise from our interactions with data systems. This necessity forms the foundation of HDI, a concept that seeks to redefine our relationship with data and data-driven technologies. 

To fully grasp the transformative potential of the HDI Framework presented in this article, it is crucial to first understand the fundamental principles and scope of HDI itself. The following section provides a comprehensive definition of Human-Data Interaction, setting the stage for our exploration of its practical applications and far-reaching implications.



§ HUMAN-DATA INTERACTION DEFINITION

Human-Data Interaction (HDI) is an emerging field that seeks to place the human at the center of the data flows and processes that characterize our digital world. As defined by Mortier et al. (2014), HDI focuses on the interactions between individuals and data systems, emphasizing the need for these systems to be designed with human values and agency in mind <cit.>. HDI extends beyond traditional human-computer interaction by specifically addressing the challenges posed by ubiquitous data collection, processing, and use in modern society.
The core principles of HDI, as initially proposed by Mortier et al. and further developed by subsequent researchers, are:


    
  * Data Legibility: This principle emphasizes the importance of making data processes and systems understandable to individuals. As Crabtree and Mortier (2015) argue, data legibility involves not just transparency, but also the provision of tools and interfaces that allow people to comprehend how their data is collected, analyzed, and used <cit.>. This includes clear communication about data practices and the potential implications of data use.
    
  * Agency: The principle of agency focuses on empowering individuals to take action regarding their personal data. This goes beyond mere consent to data collection, advocating meaningful control over data throughout its lifecycle. Luger et al. (2015) highlight that agency in HDI contexts involves giving individuals the ability to make informed decisions about their data, including the right to access, modify, and delete personal information <cit.>.
    
  * Negotiability: This principle recognizes that the context and meaning of data can change over time and across different situations. As such, HDI advocates for systems that allow for ongoing negotiation of data use and permissions. Coles-Kemp et al. (2018) emphasize that negotiability is crucial for addressing the dynamic nature of privacy preferences and the evolving relationships between individuals and data-driven services <cit.>.
 
These core principles are interconnected and mutually reinforcing. For instance, data legibility is a prerequisite for meaningful agency, as individuals can only make informed decisions about their data if they understand how it is being used. Similarly, negotiability supports both legibility and agency by allowing for adaptable and context-sensitive data relationships.

Recent studies have expanded on these core principles. For example, Victorelli et al. (2020) propose additional dimensions to the HDI framework, including data quality, security, and ethical use <cit.>. Their work underscores the evolving nature of HDI and its responsiveness to emerging challenges in the data-driven landscape.

Furthermore, Sailaja et al. (2019) have applied HDI principles to specific domains such as media experiences, demonstrating how these concepts can be operationalized in real-world contexts <cit.>. Their research highlights the practical implications of HDI and its potential to transform user experiences in data-intensive environments.

By defining and elaborating on these core principles, HDI provides a comprehensive framework for addressing the complex challenges of data ethics, privacy, and user empowerment in the digital age. It offers a foundation for developing more human-centric data practices and technologies that respect individual rights while harnessing the potential of data-driven innovation.

The multifaceted nature of Human-Data Interaction, as defined above, underscores the complexity of the challenges it aims to address. These challenges span across technological, ethical, legal, and social domains, highlighting the limitations of singular, discipline-specific approaches. 

To fully realize the potential of HDI and effectively implement its principles, we must draw upon diverse fields of expertise and foster collaboration across traditional boundaries. This necessity for a holistic perspective leads us to explore the critical importance of interdisciplinary approaches in HDI. By integrating insights from various disciplines, we can develop more comprehensive and effective solutions to the intricate problems posed by our increasingly data-driven world.



§ IMPORTANCE OF INTERDISCIPLINARY APPROACHES

The HDI Framework underscores the necessity of employing interdisciplinary approaches to develop comprehensive and effective solutions for the challenges posed by the increasing leverage of personal data <cit.>. By integrating perspectives from diverse fields, the framework can harness the complementary expertise and insights required to address the multifaceted nature of human-data interactions.For instance, human-computer interaction can provide valuable insights into user needs, preferences, and behaviors, while data science can offer technical expertise in data manipulation, analysis, and modeling. 

Additionally, fields such as ethics and policy can contribute to the development of ethical frameworks, governance structures, and regulatory mechanisms that ensure the responsible and trustworthy use of personal data <cit.>

The framework also seeks to identify and mitigate the socio-technical challenges that arise from the increasing leverage of personal data in various domains, such as privacy concerns, data transparency and control, accountability issues, and lack of trust.These challenges have been studied and discussed in various disciplines, but effective responses that alleviate them are yet to be fully realized <cit.>

The proposed framework takes an interdisciplinary approach, integrating perspectives from fields such as human-computer interaction, data science, ethics, and policy to develop comprehensive and effective solutions <cit.>.By addressing these socio-technical challenges, the framework aims to create data-driven experiences and systems that are aligned with user needs and values, ultimately fostering a more equitable and user-centric data-driven landscape.<cit.>

The multifaceted nature of Human-Data Interaction necessitates an interdisciplinary approach, drawing insights from diverse fields such as computer science, ethics, law, and social sciences. This interdisciplinary perspective becomes particularly crucial when considering the complex and dynamic nature of the data lifecycle. As data moves through its various stages from collection and processing to storage, utilization, and eventual disposal it intersects with a wide array of disciplines and stakeholders. 

The integration of diverse expertise ensures that each phase of the data lifecycle is approached with a comprehensive understanding of its technical, ethical, and societal implications. For instance, while computer scientists might focus on secure data storage methods, ethicists can provide insights on fair data use, and legal experts can ensure compliance with data protection regulations. By bridging the gap between these disciplines, the HDI framework can more effectively address the challenges that arise at each stage of the data lifecycle, fostering a more holistic and human-centric approach to data management.



§ THE DATA LIFECYCLE

The Data Lifecycle is a critical component of the HDI Framework as it encapsulates the journey of data from its creation to its eventual disposal. This holistic view ensures that ethical considerations and human-centric practices are applied at every stage of data handling. In the contemporary landscape, data pervades every aspect of our society, influencing diverse domains from personalized service offerings to strategic policy-making on a global scale. The utilization of algorithms, powered by this extensive data, is pivotal in sculpting our daily interactions. They undertake tasks of substantial complexity and consequence, from analyzing medical imagery to curating news feeds, thereby making decisions that carry significant ramifications for society<cit.>. Yet, the operationalization of these data-driven frameworks remains an intricate challenge, as they must grapple with the dynamic and relational nature of data itself. 

As data moves through the stages of the lifecycle, from gathering and organizing to analysis and application, it becomes enmeshed with multifaceted social, ethical, and political implications<cit.>.The HDI Framework seeks to address these complexities by placing the human at the center of the data lifecycle, ensuring that the design, deployment, and governance of data-driven systems are aligned with user needs,preferences, and values.

Despite the undeniable influence of big data, there exists a peril of neglecting the individual lives encapsulated within these datasets, lives that are invariably affected by the operational systems<cit.>. While these systems have the potential to yield positive outcomes, the "tyranny of data" can also lead to violations of privacy, information asymmetry, lack of transparency, discrimination, and social exclusion <cit.>

To mitigate these risks, the proposed HDI Framework advocates for a comprehensive understanding of the data lifecycle, encompassing the processes of collection, storage, processing, and utilization <cit.>By examining each stage of this lifecycle through the lens of human values, needs, and preferences, the framework aims to develop solutions that empower individuals and foster trust in data-driven decision-making <cit.>.

In reaction to this oversight, the domain of HDI has been proposed, advocating for a data environment that is ethical and empowering. Within this framework, individuals are regarded not as mere subjects of data extraction but as active agents endowed with rights to agency, transparency, and safeguards against data misuse <cit.>. This discourse posits that the creation of a data ecosystem centered around human interests is not merely a moral obligation but is also essential for sustaining trust and legitimacy in the digital era. 

To this end, the proposed framework delineates the data lifecycle through the lens of human-centric values, encompassing the following key phases:


  * Data collection: Ensuring informed consent and transparency around the purpose and use of collected data <cit.>.
  * 

Data processing: Enabling users to understand the algorithms and inferences applied to their data, as well as the ability to contest and rectify inaccuracies <cit.>.
  * 

Data storage and management: Providing users with control over the retention and deletion of their personal information, as well as mechanisms for data portability <cit.>.
  * . 

Data utilization: Empowering users to comprehend how their data is being leveraged and to exercise agency in determining the scope and context of its use <cit.>.
  * .

Data sharing and exchange: Affording users the right to selectively share their data and to revoke access as needed, fostering a sense of data ownership and control <cit.>.
  * 

Data disposal: Ensuring the secure and responsible deletion of user data upon request or when it is no longer needed, in alignment with privacy regulations and user preferences <cit.>.
  * 

Data analytics and insights: Enabling users to understand the inferences and predictions made about them, and to dispute or correct any erroneous conclusions <cit.>.
  * 

Ongoing monitoring and evaluation: Establishing mechanisms for continuous assessment and improvement of the data ecosystem, addressing evolving user needs and emerging technological advancements <cit.>.


The HDI Framework proposes a comprehensive, human-centric approach to the data lifecycle, aiming to instill individual empowerment and collective accountability in our data-driven landscape <cit.>. This paradigm shift goes beyond mere technical solutions, outlining a structured approach for embedding ethical principles throughout the entire data journey.

Central to this framework is the development of stringent policies upholding informed consent and data sovereignty, the integration of user-empowerment tools, and the promotion of data ethics education <cit.>. Despite the complexities of modern data systems and the risk of algorithmic bias <cit.>, the adoption of HDI principles offers profound benefits, positioning organizations to utilize data responsibly, foster innovation, and contribute to societal well-being.

The HDI Framework represents a transformative shift, placing individuals at the core of shaping our data-driven future. By prioritizing user agency, transparency, and ethics, it lays the groundwork for a more equitable and empowering data ecosystem <cit.>.

Human engagement is crucial throughout the data lifecycle to ensure ethical, equitable, and socially beneficial data practices <cit.>. This involves empowering individuals with agency and choice in data collection, processing, and utilization, while fostering transparency and accountability in data-driven systems.
At the collection stage, individuals should receive comprehensive information about the purpose, scope, and intended use of their data <cit.>. This transparency enables informed consent and allows for selective data sharing based on personal preferences.
During processing and analysis, individuals must be empowered to understand the algorithms and processes applied to their data, including potential implications. This legibility allows users to contest inaccuracies and understand how their data is being used <cit.>.

In storage and management, individuals should have control over data retention and deletion, with data portability features allowing seamless transfer between service providers <cit.>.The utilization and sharing phases require balancing individual privacy with potential societal benefits from data-driven insights. Individuals should be able to selectively share data while maintaining control over access and scope.
By embedding human agency, transparency, and control throughout the data lifecycle, the HDI Framework fosters a more equitable and empowering data ecosystem <cit.>. 
This approach safeguards individual rights while incentivizing responsible organizational data practices.
Crucially, the application of user data in decision-making must include mechanisms for user oversight and the ability to challenge decisions, empowering individuals to shape how their information is used and fostering accountability and transparency <cit.>. 

Incorporating this lifecycle approach into the HDI Framework ensures consistent application of human-centric principles across all data-related activities, enhancing both ethical practices and overall data management efficiency.

While the Data Lifecycle provides a comprehensive framework for managing data ethically throughout its existence, implementing such a structure requires careful planning and strategic considerations. As organizations prepare to embed HDI principles across the data lifecycle, they must first address several preliminary considerations. 

These foundational steps ensure that the organization is well-positioned to successfully integrate HDI practices into every stage of the data journey. By thoughtfully examining these preliminary considerations, organizations can create a solid groundwork for implementing a human-centric approach to data management that aligns with ethical standards, regulatory requirements, and stakeholder expectations. 

The following section outlines these crucial preliminary considerations, which serve as a bridge between understanding the theoretical importance of the data lifecycle and practically implementing HDI principles within an organization's unique context.




§ PRELIMINARY CONSIDERATIONS


Before implementing a Human-Data Interaction (HDI) framework, organizations must carefully evaluate and prepare their existing structures, processes, and culture. These preliminary considerations serve as a crucial foundation, ensuring that the subsequent implementation of HDI principles is well-informed, strategically sound, and aligned with both organizational values and ethical data practices. By thoroughly addressing these initial aspects, organizations can pave the way for a more successful and sustainable integration of HDI principles into their core operations. The following subsections outline key areas that demand attention in this preparatory phase. This stage involves laying the groundwork for implementing the insights gleaned from alignment analysis, focusing on ethical, technical, and operational considerations to improve data practices. This proactive approach ensures that the subsequent steps taken are informed, deliberate, and aligned with both organizational values and HDI principles.



 §.§ Ethical Considerations

At the forefront of preliminary considerations is the emphasis on ethics. Organizations must deliberate on the ethical implications of their data practices, particularly in terms of respecting user autonomy, ensuring privacy, and preventing harm<cit.>.This involves considering how data is collected, used, and shared, ensuring that these practices are not only legally compliant but also ethically sound. The work of Shilton<cit.> underscores the importance of embedding ethical considerations into the fabric of organizational culture and decision-making processes, ensuring that data practices reflect a commitment to respect, fairness, and responsibility toward individuals whose data are being handled. 

To this end, organizations should establish a cross-functional ethics review board or advisory panel, tasked with evaluating data-driven initiatives through the lens of HDI principles.




 §.§ Technical Considerations

Technical considerations form the backbone of effective HDI implementation. Organizations must assess the capabilities and limitations of their existing technological infrastructure, identify areas for improvement, and ensure that data practices are underpinned by robust, user-centric systems.

This may involve enhancing data management and governance frameworks, implementing user-friendly data access and control interfaces, and deploying advanced analytics and visualization tools that enhance data legibility and transparency. 

Furthermore, organizations should explore the potential of emerging technologies, such as blockchain and secure multi-party computation, to bolster data security, provenance, and user control <cit.>

Organizations must consider investments in technologies that enable more granular consent mechanisms, robust encryption, and secure data storage and transmission protocols<cit.>. In addition, the development or adoption of tools for bias detection and mitigation in algorithms, as well as those that enhance the explainability of machine learning models, is critical. These technical upgrades and innovations are essential to operationalize the HDI principles in a way that is both effective and sustainable.




 §.§ Operational Considerations

Operational considerations encompass the processes and policies that govern data practices within the organization. This includes re-visiting data governance frameworks to explicitly incorporate HDI principles, ensuring that data collection, storage, use, sharing, and disposal policies are aligned with ethical standards and technical capabilities<cit.>. The establishment of clear, transparent communication channels for stakeholders, including data subjects, is crucial for fostering trust and accountability. 

Operational considerations also involve the training and engagement of employees across the organization, from leadership to frontline staff, in understanding and championing HDI principles. The work of Burr et al.<cit.> highlights the importance of integrating legal, technical, and policy expertise in crafting and implementing data practices that are not only compliant but also ethically and socially responsible.




 §.§ Stakeholder Engagement

Engaging with stakeholders, including users, regulatory bodies, and civil society organizations, is a critical component of preliminary considerations. This engagement ensures that the organization's efforts to implement HDI principles are responsive to the needs and concerns of those affected by its data practices<cit.>. Through workshops, focus groups, and other participatory methods, organizations can gain valuable insights into user expectations, preferences, and concerns regarding data management.

User participation, in particular, is essential to understand the unique needs, preferences, and concerns of individuals whose data are being collected and used. This may involve conducting user research, collecting feedback on data practices, and establishing mechanisms for ongoing dialogue and collaboration<cit.>.Engagement with regulatory bodies and civil society organizations can provide valuable insights into emerging trends, best practices, and potential pitfalls in HDI implementation, thus guiding the organization's efforts and ensuring compliance with relevant laws and regulations.

Stakeholder feedback can provide valuable insights into potential ethical dilemmas, privacy concerns, and areas for improvement in transparency and accountability. Furthermore, this engagement fosters a collaborative approach to data governance, improving the legitimacy and effectiveness of the organization's data practices.




 §.§ Alignment Analysis

The alignment analysis phase involves a comprehensive assessment of the organization's existing data practices against the principles of HDI. This analysis serves as a foundational step in identifying areas of alignment, gaps, and opportunities for improvement, forming the subsequent implementation of HDI-centric initiatives <cit.>.

The analysis should consider the organization's data ecosystem holistically, examining factors such as data collection, storage, usage, sharing, and disposal practices. Additionally, the analysis should assess the organization's technological capabilities, governance frameworks, and stakeholder engagement mechanisms to identify areas of alignment and opportunities for enhancement.

Mapping an organization's data practices to HDI principles begins with a comprehensive review of current policies, procedures, and technical systems related to data management. This review should consider factors such as the organization's data collection methods, storage and processing practices, data sharing and disclosure protocols, and the mechanisms in place for user control and transparency. By conducting a thorough alignment analysis, organizations can gain a clear understanding of their current state and develop a roadmap to effectively implement the HDI principles.



 §.§ Audit Existing Practices

The initiation of a HDI framework within an organization necessitates a foundational step: a meticulous audit of existing data handling practices. This critical examination serves not merely as a procedural checkpoint but as a profound inquiry into the organization's data lifecycle management, from inception to obsolescence.

By undertaking this audit, an organization can delineate a comprehensive baseline from which to enhance and refine its data stewardship, guided by the tenets of HDI.


  * 

Comprehensive Data Mapping: The audit commences with an exhaustive mapping of the data lifecycle, illustrating the flow of data from its collection points through to its final disposition. This visual documentation encompasses identifying all data ingress points, such as sensors, web interfaces, and third-party integrations, and tracing the data's journey across storage, processing, internal sharing, and external disclosures. Humayun et al.<cit.> underscore the importance of understanding these data flows, highlighting how such mapping can unveil the intricacies and potential vulnerabilities within an organization's data ecosystem.
  * 

Policies and Procedures Analysis: A thorough review of the existing data governance frameworks is imperative. This includes scrutinizing privacy policies, data handling procedures, and terms of service to ascertain their alignment with HDI principles such as informed consent, data minimization, and user rights<cit.>. The clarity, accessibility, and comprehensiveness of these documents are evaluated to ensure they serve not only as legal compliance mechanisms but also as transparent guides for users.
  * 

Security Measures Evaluation: Assessing the robustness of data security measures forms a critical part of the audit. Organizations must evaluate their encryption protocols, access control mechanisms, and incident response strategies to identify vulnerabilities<cit.>. This assessment helps in pinpointing areas where enhancements are necessary to fortify the organization against data breaches and unauthorized access.
  * 

Data-Driven Decision-Making Scrutiny: The use of data in decision-making processes, particularly when involving algorithms or machine learning models, requires rigorous examination. The audit should investigate these models for biases and their explainability, ensuring that decision-making processes are transparent and equitable<cit.>. This scrutiny aims to guarantee that human oversight is effectively integrated, allowing for intervention and accountability.
  * 

User Interface Inspection: Finally, the interfaces through which users interact with the organization's data systems warrant careful evaluation. This involves assessing the clarity and efficacy of consent mechanisms, the ease with which users can access or amend their data, and the transparency surrounding the usage of their data <cit.>


This auditing process, necessitating a collaborative effort among diverse organizational teams, embodies a multidisciplinary approach. By integrating insights from technical staff, legal experts, and user experience designers, organizations can gain a holistic understanding of their current data handling practices.<cit.>

Such a comprehensive audit not only illuminates existing strengths and weaknesses but also paves the way for the development of a nuanced HDI framework. Through targeted interventions, grounded in the insights garnered from the audit, organizations can align their data practices with core values and goals, fostering a culture of ethical data stewardship and enhancing trust among users.

Having examined these crucial preliminary considerations, organizations can lay a solid foundation for implementing a comprehensive HDI framework. These initial steps provide the necessary context and groundwork for a more detailed and structured approach to integrating HDI principles throughout an organization's data practices. 

Building upon this preparatory phase, we now turn to a detailed exploration of the HDI Framework Schema. This schema offers a comprehensive, process-based model that operationalizes HDI principles, providing organizations with a practical guide to embed ethical data practices into their core operations. By progressing from preliminary considerations to this structured framework, organizations can systematically transform their approach to data management, ensuring that human values and ethical considerations are central to all data-related activities.




§ HDI FRAMEWORK SCHEMA

The proposed HDI framework is a comprehensive process-based model designed to address the complex challenges of data management and utilization. Building upon the foundational, this framework expands and operationalizes the HDI principles to provide a practical guide for organizations, policymakers, and researchers.

The framework is structured around nine key components, emphasizing its process-oriented nature:


    
  * Foundational Principles
    
    
  * Agency: Empowering individuals with control over their data.
    
  * Legibility: Making data processes transparent and understandable.
    
  * Equity Ensuring fair and unbiased data practices.
    
  * Accountability: Clearly defining responsibility for data use.
    
  * Privacy and Data Protection: Safeguarding personal information.
    
  * Participation and Engagement: Meaningful stakeholder involvement throughout the data lifecycle, from design to deployment.
    
  * Negotiability: Allowing flexible, context-based data permissions.
    
  * Human-Centricity: Prioritizing human needs in data systems.
    
  * Consent: Ensuring informed and meaningful consent for data collection and use.
    
    
    
  * Structural Components
    
    
  * Data Governance: Establishing policies for ethical data management.
    
  * Technological Infrastructure: Implementing tools supporting HDI principles.
    
  * Organizational Culture: Fostering an ethical data mindset.
    
  * Stakeholder Engagement: Involving users in data decisions.
    
  * Ethical Review Processes: Continuously assessing data practices.
    
  * Data Ethics Board: Establishing a dedicated group to oversee ethical data practices.
    
  * Data Quality Management: Ensuring data accuracy and relevance.
    

    
  * Ethical Considerations
    
        
  * Respect for Individual Autonomy: Honoring individual choices.
        
  * Equity and Justice: Promoting fairness in data practices.
        
  * Transparency: Ensuring openness in data processes.
        
  * Responsible Innovation: Ethically advancing data technologies.
        
  * Environmental Impact: Considering sustainability in data management.
        
  * Bias Mitigation: Addressing prejudices in data and algorithms.
        
  * Ethical AI Use: Aligning AI with HDI principles.
        
  * Cultural Sensitivity: Considering diverse cultural perspectives in data practices.
    
    
    
  * Practical Considerations
    
        
  * System Integration: Incorporating HDI into existing infrastructures.
        
  *  Technological Feasibility: Assessing implementability of HDI solutions.
        
  * Scalability: Ensuring HDI practices can grow with the organization.
        
  * Regulatory Compliance: Meeting legal data requirements.
       
  *  Organizational Capacity: Evaluating readiness for HDI adoption.
        
  * Education and Training: Preparing staff for HDI implementation.
        
  * Cost-Benefit Analysis: Assessing financial implications of HDI.
        
  * Risk Management: Identifying and mitigating data-related risks.
        
  * Interoperability: Ensuring HDI practices work across different systems and platforms.
    
    
    
    
  * Implementation Roadmap 
    
        
  * Assessment:  Evaluating current data practices against HDI principles. 
        
  * Design:  Developing HDI-aligned systems and processes.
        
  * Integration:  Implementing HDI practices across the organization.
        
  * Evaluation: Continuous monitoring and improvement of HDI initiatives.
        
  * Adaptation: Updating practices for emerging challenges.
        
  * Pilot Testing: Implementing HDI practices in small-scale projects before full deployment.
    
    
    
  * Enhancing HDI Culture
    
        
  * Employee Training Programs: Educating staff on HDI principles.
        
  * Stakeholder Education: Broadening the scope to include all affected parties, not just direct users.
        
  * Continuous Learning and Adaptation: Updating training based on new trends.
        
  * Cross-functional Collaboration: Fostering interdepartmental HDI understanding.
    
    
    
  * Data Lifecycle Management
    
        
  * Collection: Gathering data ethically and transparently.
        
  * Processing: Analyzing data fairly and accountably.
        
  * Storage: Retaining data securely and responsibly.
        
  * Utilization: Using data ethically in decision-making.
        
  * Sharing: Exchanging data with consent and control.
        
  * Disposal: Deleting data properly and timely.
    

    
  * Monitoring and Adaptation
    
        
  * Continuous Assessment: Regularly evaluating HDI practices.
        
  * Feedback Mechanisms: Gathering stakeholder input on data practices.
        
  * Adaptive Governance: Evolving policies with changing needs.
        
  * Emerging Technology Integration: Incorporating new HDI-enhancing tools.
        
  * Performance Metrics: Tracking HDI-specific key performance indicators.
       
  *  External Audits: Conducting independent assessments of HDI practices.
    
        
    
  * Complex and dynamic considerations
    
        
  * Complexity: Acknowledging the intricate nature of data systems and the need for nuanced, context-specific HDI implementations.
        
  * Evolving Technologies: Addressing the continual emergence of new data technologies and their implications for HDI.
        
  * Regulatory Landscape: Keeping pace with changing data protection laws and regulations across different jurisdictions.
        
  * Societal Shifts: Recognizing changing public attitudes towards data privacy and use.
        
  * Ethical Dilemmas: Preparing for unforeseen ethical challenges that may arise with new data uses.
        
  * Cross-border Data Flows: Managing HDI principles in the context of global data transfer and varying international standards.
        
  * Balancing Innovation and Ethics: Navigating the tension between rapid technological advancement and ethical considerations.
        
  * Adaptive Strategies: Developing flexible approaches that can evolve with changing data ecosystems.
    
    


This process-based framework is designed to be iterative and adaptable across various sectors and organizational sizes. It recognizes the dynamic nature of the data landscape, as highlighted by <cit.>, and provides flexibility to address emerging challenges.

The framework incorporates insights from recent research on algorithmic accountability <cit.> and privacy economics <cit.>, ensuring a holistic approach to data ethics and management. It emphasizes the importance of continuous learning and adaptation, acknowledging that HDI implementation is an ongoing process rather than a one-time initiative.

By integrating these components into a cohesive, process-oriented approach, the proposed HDI framework offers a roadmap for organizations to navigate the complexities of data ethics, privacy, and user empowerment. It aims to foster a data ecosystem that not only drives innovation and efficiency but also respects human values and promotes societal well-being.

This framework represents a significant step towards realizing the vision of a human-centric data society, where technological advancements and ethical considerations evolve in tandem, as advocated <cit.> in their work on AI ethics. Its process-based nature ensures that organizations can continually refine and improve their HDI practices in response to new challenges and opportunities in the ever-evolving data landscape.



§ FOUNDATIONAL PRINCIPLES

The HDI Framework presents a comprehensive and multidimensional approach to managing and utilizing data in a manner that prioritizes ethical considerations, upholds fundamental human rights, and promotes the overall well-being of society. This holistic framework guides organizations in cultivating a data ecosystem that deeply respects individual autonomy, empowering users with meaningful control over their personal information, including its collection, usage, sharing, and deletion.<cit.>

The framework emphasizes the critical importance of transparency, requiring clear and comprehensible explanations regarding data handling practices, algorithmic decision-making processes, and the security protocols in place to safeguard data integrity<cit.>. Ensuring fairness and non-discrimination is a core tenet, demanding proactive efforts to identify and mitigate biases within datasets and algorithmic systems, thereby promoting equitable outcomes for all individuals. Accountability is woven throughout the framework, with mechanisms established to address instances of data misuse, facilitate redress for affected parties, and demonstrate organizational responsibility. 

Additionally, the framework underscores the value of meaningful stakeholder engagement, advocating for the inclusion of diverse perspectives and the co-creation of ethical data practices with users and affected communities. By outlining these guiding principles, structural components, implementation strategies, and multifaceted considerations<cit.>, the HDI Framework provides a comprehensive blueprint for organizations to embed ethical practice and user-centric values at the very core of their data management and data-driven decision-making processes.




 §.§ Agency

Human agency and autonomy are central to the HDI framework, emphasizing the empowerment of individuals in their interactions with data-driven systems. This principle advocates for giving individuals control over their data, including its collection, use, sharing, and deletion. It underscores the importance of enabling people to make informed decisions about their data, reflecting a commitment to respecting personal autonomy and privacy <cit.>

Individuals have the right to exercise control over their personal data, including the right to access, modify, and delete their information, as well as to make informed decisions about data collection and use. Respecting autonomy acknowledges individuals as active agents with the capacity to make choices about their data. This principle is fundamental to ensuring that data practices empower rather than undermine user agency.<cit.>




 §.§ Legibility

Transparency and explainability within HDI demand clear, understandable information about data practices, the workings of algorithms, and the security measures in place to protect data<cit.>. This principle is crucial for building trust and accountability, allowing users to grasp how their data is processed and how decisions that affect them are made. Explainable AI (XAI) systems play a pivotal role in demystifying algorithmic processes, making technology accessible and comprehensible to non-experts (Ribeiro, Singh, And Guestrin, 2016).<cit.>.

Maintain openness about data practices, including the collection, use, and sharing of data, and implement mechanisms for accountability in decision-making processes. Transparency and accountability are vital for building trust, facilitating oversight, and enabling individuals to understand and challenge decisions made with or about their data <cit.>




 §.§ Equity
 
Fairness and non-discrimination call for proactive efforts to detect and mitigate biases in datasets and algorithms, ensuring equitable outcomes for all users<cit.>. This principle recognizes the potential for data-driven systems to perpetuate or even exacerbate existing inequalities. Research in algorithmic fairness seeks to develop methodologies and tools to identify and correct biases, promoting fairness in automated decision-making processes <cit.>

Ensure equitable access to the benefits of data technologies and protect against discriminatory practices and biases that may arise from data collection, analysis, and application. Addressing justice and equity is crucial for mitigating the risk of exacerbating existing inequalities and ensuring that the benefits of data technologies are shared across all sectors of society <cit.>




 §.§ Accountability

Accountability and redress in the HDI framework ensure that mechanisms are in place to address any misuse of data, provide remedies to affected individuals, and demonstrate organizational responsibility<cit.>. This principle involves establishing clear policies and procedures for data governance, including how data breaches and privacy violations are handled. It also encompasses the need for regulatory compliance and the ability to audit and review data practices to uphold ethical standards <cit.>

Data practices should aim to benefit individuals and communities while minimizing harm and preventing misuse of data that could lead to adverse outcomes. The dual obligation to do good and avoid harm is a cornerstone of ethical practice, emphasizing the responsibility of data handlers to consider the broader impact of their actions on society.<cit.>




 §.§ Privacy and Data Protection

Privacy and data protection are foundational to HDI, safeguarding individuals' rights to control their personal information and ensuring that data is handled securely and respectfully<cit.>. This principle is aligned with global privacy regulations, such as the General Data Protection Regulation (GDPR) in the European Union, which set standards for data collection, processing, and storage. Implementing privacy-by-design approaches and privacy-enhancing technologies (PETs) are practical steps towards operationalizing this principle.

Uphold individuals' rights to privacy and data protection, implementing secure and responsible data handling practices to safeguard personal information from unauthorized access and exploitation.

Privacy is essential for personal security, freedom of expression, and protection against surveillance and control, making it a fundamental aspect of ethical data practices.




 §.§ Participation and Engagement

Participation and engagement advocate for the inclusion of stakeholders, including data subjects, in the design and governance of data-driven systems<cit.>. This principle emphasizes the value of diverse perspectives and the importance of co-creating ethical data practices with users and affected communities. It supports the idea of democratizing data, where individuals and communities have a say in how data about them is used and for what purposes.

Foster inclusive and participatory approaches that engage stakeholders, particularly those most affected by data practices, in the governance, design, and implementation of data technologies. Participation and empowerment ensure that diverse perspectives are considered, supporting democratic values and enhancing the relevance and acceptability of data initiatives. Engaging with users and impacted communities helps to ensure that data practices are responsive to their needs and concerns, promoting transparency, trust, and shared responsibility <cit.>

Implementing these foundational principles requires a multidisciplinary approach, combining technical solutions with ethical, legal, and social insights. The HDI framework aims to foster a data ecosystem that respects individual rights and promotes social good, ensuring that technology serves humanity in equitable and responsible ways.



 §.§ Negotiability

The principle of negotiability in HDI recognizes that the context and meaning of data can change over time and across different situations. It emphasizes the need for flexible, dynamic approaches to data permissions and use. Negotiability allows for ongoing adjustments to data sharing preferences, acknowledging that user priorities and circumstances may evolve <cit.>.
Implementing negotiability involves:



  * Developing mechanisms for users to revise their data sharing preferences over time.

  * Creating systems that allow for context-dependent data permissions.

  * Establishing protocols for renegotiating data use agreements as new uses or insights emerge.


Mortier et al. argue that negotiability is crucial for addressing the dynamic nature of privacy preferences and the evolving relationships between individuals and data-driven services <cit.>. This principle aligns with the concept of "contextual integrity" proposed by Nissenbaum, which posits that the appropriateness of information flows depends on context-specific norms <cit.>.



 §.§ Human-Centricity

Human-centricity places human needs, values, and experiences at the core of data system design and operation. This principle ensures that technological advancements serve human interests rather than subordinating human agency to algorithmic imperatives <cit.>.

Key aspects of human-centricity in HDI include:



  * Prioritizing user experience and accessibility in data interfaces.

  * Considering the psychological and social impacts of data practices on individuals and communities.

  * Ensuring that data-driven systems augment rather than replace human decision-making capacities.


Shneiderman advocates for a human-centered approach to AI and data systems, emphasizing the importance of keeping humans "in the loop" and designing systems that enhance human capabilities <cit.>. This aligns with the broader movement towards "human-centered AI" that seeks to develop technologies that are beneficial, comprehensible, and controllable by humans <cit.>.



 §.§ Consent

Informed consent is a foundational principle in HDI, rooted in respect for individual autonomy and self-determination. It requires that individuals be provided with clear, understandable information about data collection, use, and potential implications before agreeing to share their data <cit.>.
Effective implementation of consent in HDI frameworks involves:



  * Developing clear, accessible consent mechanisms that go beyond mere legal compliance.

  * Providing granular options for consent, allowing users to selectively agree to specific data uses.

  * Ensuring that consent is ongoing and revocable, with easy-to-use tools for withdrawing or modifying consent.

  * Educating users about the implications of their consent decisions.


Recent research has highlighted the limitations of traditional notice and consent models in the big data era, leading to proposals for more dynamic and user-centric approaches to consent <cit.>. These new models aim to address the challenges of informed consent in complex, rapidly evolving data ecosystems.

The foundational principles of Human-Data Interaction (HDI) outlined above—agency, legibility, equity, accountability, privacy, participation, negotiability, human-centricity, and consent—provide the ethical and conceptual framework upon which robust HDI practices are built. However, translating these principles into practical, operational realities within organizations requires a set of concrete structural components. 

These components serve as the architectural elements that enable the realization of HDI principles in real-world data ecosystems. As we transition from the theoretical underpinnings to the practical implementation of HDI, it becomes crucial to examine how these foundational principles can be embedded into organizational structures, technological systems, and governance frameworks. 

The following section on Structural Components explores the tangible mechanisms and processes that organizations can adopt to operationalize HDI principles, ensuring that ethical considerations are not merely aspirational but are deeply integrated into the fabric of data management and utilization practices. By bridging the conceptual with the practical, we can begin to envision how HDI can transform from a set of ideals into a lived reality within complex organizational environments.




§ STRUCTURAL COMPONENTS

The structural components of the HDI framework provide a tangible architecture for embedding ethical principles into the practical operations of data-driven systems. These components are critical for translating ethical commitments into actionable strategies and processes that can be systematically applied within organizations. This section outlines the key structural elements essential for establishing a robust HDI practice, drawing on interdisciplinary research to support their implementation.



 §.§ Data Governance

A comprehensive data governance is essential for ensuring that data management practices align with ethical principles and regulatory requirements. This framework encompasses policies, standards, and procedures that guide how data is collected, stored, accessed, and used within an organization.


  * 

Policies and Protocols: Develop and implement clear policies that cover informed consent, data minimization, purpose limitation, data quality, retention, and deletion. These policies should also address security measures and incident response plans to protect data integrity and confidentiality.<cit.>
  * 

Audit and Compliance: Establish regular audit processes to evaluate the alignment of data practices with HDI principles, legal standards, and evolving societal expectations<cit.>. Compliance checks should ensure adherence to internal policies and external regulations, facilitating accountability and continuous improvement <cit.>



  * 

Roles and Responsibilities: Define clear roles and responsibilities for data stewardship, including data owners, data stewards, and data governance committees. These roles should be empowered to make decisions, enforce policies, and address issues that arise in the course of data management.<cit.>





 §.§ Technological Infrastructure

Technological tools and systems play a crucial role in facilitating ethical data practices. Using technology, organizations can improve transparency, protect privacy, and empower users.


  * 

Consent Management Platforms (CMPs): Deploy CMPs to streamline the process of obtaining and managing user consent for data collection and processing, ensuring that consent is informed, specific, and revocable <cit.>
  * 

Data Visualization Tools: Utilize data visualization tools to make data practices more transparent and comprehensible to non-expert stakeholders, aiding in the communication of complex information.<cit.>
  * 

Explainable AI (XAI) Systems: Implement XAI systems to provide clear explanations of algorithmic decisions, fostering transparency and trust in automated processes<cit.>.
  * 

Privacy-Enhancing Technologies (PETs): Incorporate PETs to minimize personal data exposure and enhance data security, supporting privacy by design and by default principles<cit.>.





 §.§ Organizational Culture

At the heart of ethical HDI are practices that prioritize human values and well-being. This involves training, stakeholder engagement, and the creation of channels for feedback and participation.


  * 

Training and Development: Implement comprehensive training programs for staff that cover data ethics, privacy, security, and bias mitigation. This training should be tailored to different roles within the organization and updated regularly to reflect new developments and insights <cit.>
  * 

 User Education and Empowerment: Develop initiatives to educate users and affected communities about their data rights and how to exercise them. This includes providing accessible resources and support for understanding data practices and the implications of data decisions.<cit.>




 §.§ Stakeholder Engagement:
 Create systems for gathering meaningful input and feedback from a diverse range of stakeholders, such as employees, users, and external communities. This involvement should guide the development of policies, the selection of technologies, and the implementation of operational practices, ensuring a variety of perspectives are considered <cit.>. 

Additionally, engage a wide array of stakeholders, including data subjects, employees, regulatory bodies, and civil society organizations, to understand their concerns, expectations, and viewpoints on data usage and ethical matters <cit.>. This inclusive approach ensures that different viewpoints are integrated into the HDI framework.



 §.§ Data Quality Management

The structural components of the HDI framework offer a comprehensive blueprint for organizations to navigate the complex ethical challenges inherent in data interaction. By deeply integrating these core elements into their day-to-day operations, organizations can foster a strong culture of responsibility, accountability, and transparency <cit.>. 

This, in turn, enhances trust and credibility with a diverse range of stakeholders, from employees to customers to affected communities. Ultimately, the implementation of this ethical HDI framework can enable organizations to make meaningful contributions towards building a more equitable, inclusive, and socially just digital ecosystem that prioritizes human values and well-being.

The implementation of HDR frameworks is a complex undertaking that requires meticulous attention to both ethical and practical considerations. As organisations endeavour to harmonise their data practices with the tenets of HDR, they confront a multitude of challenges and opportunities. The following section delineates pivotal ethical and practical considerations to assist organisations in the responsible integration of HDR frameworks, drawing upon interdisciplinary research to inform these concepts.




§ ETHICAL CONSIDERATIONS



  §.§.§ Respect for Individual Autonomy

Ensuring that individuals have meaningful control over their data requires robust mechanisms for informed consent and ongoing consent management. Organizations must carefully balance the need for data utility with respect for individual choice and privacy.<cit.> This involves developing dynamic consent platforms that empower users to adjust their privacy preferences as the contexts and applications of their data evolve over time.

These consent platforms should provide users with clear and accessible information about how their data will be collected, used, and shared. They should also offer granular controls that allow individuals to selectively authorize or revoke access to specific data types or use cases, rather than relying on a one-size-fits-all approach.<cit.>

By giving users the ability to monitor and manage their data preferences on an ongoing basis, organizations can foster a stronger sense of trust and transparency, while still preserving the valuable insights that can be derived from aggregated data. This balanced approach is crucial for upholding individual autonomy while also enabling the responsible and ethical use of data to drive innovation and societal progress.


  §.§.§ Equity and Justice


Data practices can inadvertently reinforce or exacerbate existing social, economic, and political inequalities. It is crucial for organizations to proactively identify and mitigate various types of biases that may be present in their data collection, analysis, and application processes <cit.>

 These biases can manifest in the form of underrepresentation of marginalized groups, flawed algorithms that amplify historical discrimination, and the exclusion of diverse perspectives during the development of data-driven systems. To address these challenges, organizations should adopt inclusive design and engagement practices that actively involve a wide range of stakeholders, including underrepresented and marginalized communities, in the development, testing, and ongoing evaluation of their data systems. 

This collaborative approach can help uncover and address blind spots, ensure that the needs and concerns of diverse users are prioritized, and ultimately prevent the perpetuation of harm to vulnerable populations. By embedding equity and justice as core principles throughout the data lifecycle, organizations can work towards building more inclusive, fair, and responsible data practices that benefit all members of society.


  §.§.§ Transparency and Accountability


Achieving transparency in complex data systems is undoubtedly challenging, but it is essential for ensuring meaningful accountability. Stakeholders, including employees, users, and the broader public, should have access to comprehensive, understandable information about how their data is collected, used, and shared by organizations<cit.>. To address this critical need, organizations must develop robust, user-friendly data policies and communication channels that clearly and transparently inform individuals about the full scope of their data practices. 

This includes providing detailed explanations of the specific purposes for which data is collected and used, as well as the concrete measures and safeguards in place to ensure responsible data management and ethical stewardship of sensitive information<cit.>.

Embracing a high degree of transparency allows organizations to build stronger trust relationships with their stakeholders, empowering individuals to make well-informed choices regarding the use of their personal data. This demonstrates the organization's commitment to upholding data privacy principles and ethical data governance practices.


  §.§.§ Responsible Innovation


While data-driven technologies offer significant potential for societal benefit, their development and deployment must be guided by strong ethical principles<cit.>. Establishing comprehensive ethical review processes and frameworks is crucial for enabling responsible innovation. These frameworks should thoroughly assess the far-reaching implications of new data-driven innovations, carefully considering their impact on human and societal well-being <cit.>

Furthermore, developing and deploying explainable AI tools and practices is essential to ensure transparency and accountability in the application of these transformative technologies. 

Enhancing the accessibility and comprehensibility of algorithmic decision-making processes for nonexpert stakeholders can enable organizations to foster increased trust and facilitate meaningful engagement with these systems. This level of openness and understanding is vital for upholding the principles of ethical data governance and ensuring that the benefits of data-driven innovation are equitably distributed throughout society <cit.>

The responsible development and application of data-driven technologies must be based on a strong commitment to the protection of ethical principles, the protection of individual rights and the promotion of greater social well-being<cit.>. Only through this holistic, principle-driven approach can organizations harness the immense potential of data-driven innovation while mitigating the risks and unintended consequences that may arise.


  §.§.§ Environmental and Social Impact 


Responsible data management requires organizations to adopt a holistic, sustainability-focused approach that considers the environmental and social impacts of their data production, storage, and processing activities. This includes carefully examining the energy consumption, emissions, and e-waste generated by their data infrastructure, as well as the broader societal implications of their data-driven systems <cit.>

Adopting a comprehensive perspective on the environmental and societal implications of their data-related activities enables organizations to formulate strategies that mitigate these impacts and foster more sustainable and ethically-grounded data management practices<cit.>. This holistic approach is essential to ensure that the advantages of data-driven innovation are balanced against the imperative to protect the planet and address the concerns of diverse stakeholder groups.




§ PRACTICAL CONSIDERATIONS



 §.§ Integration with Existing Systems


Integrating HDI principles into an organization's existing data infrastructure and practices requires careful and comprehensive planning to avoid potential disruptions and ensure seamless compatibility<cit.>. Conducting a thorough systems analysis is crucial to identify and address all potential integration challenges. 

This analysis should examine the organization's current data management workflows, technological capabilities, and existing policies and procedures. Based on the insights gained, a well-crafted, phased implementation plan can be developed, allowing for a gradual and strategic adaptation of HDI principles. This measured approach, with clearly defined milestones and feedback loops, enables the organization to mitigate risks, build organizational buy-in, and ensure a smooth transition that preserves data utility and operational continuity. 

Carefully examining the existing data environment and devising a measured, phased implementation plan enables the organization to successfully integrate HDI principles into its fundamental data practices without sacrificing efficiency or effectiveness.


  §.§.§ Technological Feasibility


Implementing HDI frameworks may necessitate the development or adoption of novel technologies that can effectively address the unique requirements and complexities of data management and interaction in the modern digital landscape. This may include the integration of privacy-preserving data management tools, which employ advanced cryptographic techniques and data anonymization methods to safeguard individual privacy while preserving the utility of data for authorized purposes<cit.>. Furthermore, the implementation of dynamic consent platforms can empower users with greater control and agency over the use of their personal data, allowing them to adjust their preferences as the contexts and applications of data evolve.

Evaluating the feasibility and scalability of these technological solutions is essential to ensure their successful integration within the organization's data infrastructure and practices. This assessment should consider factors such as technical capabilities, ease of implementation, compatibility with existing systems, and the ability to adapt to changing organizational and regulatory requirements over time.<cit.> 


  §.§.§ Scalability and Flexibility


Organizations must design HDI criteria that are highly scalable and adaptable, ensuring their long-term resilience and responsiveness to evolving technologies, regulatory environments, and organizational needs. This entails developing modular frameworks with interchangeable components that can be easily updated or replaced as required, without disrupting the overall system.<cit.>

By embracing a modular architecture, organizations can imbue their HDI frameworks with the capacity to adapt rapidly to evolving circumstances. This allows the organization, for instance, to update the pertinent data privacy management components in response to the introduction of new regulatory requirements, without the need to restructure the entire HDI system.

Similarly, as new data analytics tools or interaction techniques emerge, the modular design allows for the seamless integration of these innovations, enabling the organization to stay at the forefront of technological advancements.<cit.>

This scalable and adaptable architecture is crucial for safeguarding the organization's long-term viability and ensuring that its HDI practices remain aligned with evolving needs and best practices. It empowers the organization to proactively anticipate and respond to change, rather than being beholden to rigid, inflexible systems that quickly become obsolete. By prioritizing modularity and adaptability in the design of their HDI frameworks, organizations can future-proof their data practices and maintain a competitive edge in an ever-changing digital landscape.


  §.§.§ Education and Training


Successful integration of HDI principles necessitates a well-versed and empowered workforce capable of navigating the intricacies of contemporary data systems while upholding ethical data practices. Accordingly, organizations should develop comprehensive training programs for all employees, emphasizing the ethical considerations underlying their work and providing ongoing education on emerging data-related practices and technologies<cit.>. These training programs should not only impart technical knowledge but also foster a deep understanding of the ethical implications of data use, data privacy, and algorithmic decision-making.

Equipping employees with the necessary skillsets and ethical knowledge empowers them to make well-informed, values-aligned decisions when interacting with complex data systems. Additionally, these training initiatives should be accompanied by the establishment of clear organizational policies, accountability frameworks, and channels for continuous feedback and improvement <cit.>



§ IMPLEMENTATION ROADMAP

The implementation of a comprehensive Human-Data Interaction (HDI) framework within an organization represents a paradigm shift in data management practices. This roadmap delineates a structured, yet flexible approach to embedding HDI principles into the core of organizational operations. The process is inherently iterative and adaptive, recognizing the dynamic nature of both technological advances and ethical considerations in the digital age.



 §.§ Phase 1: Assessment and Planning




  §.§.§ Comprehensive Audit and Gap Analysis


The initial phase commences with a thorough, multidimensional audit of existing data practices. This audit goes beyond simple compliance checks, delving into the ethical implications of current data handling processes. Organizations must scrutinize their entire data ecosystem, from collection points to disposal mechanisms, evaluating each stage against HDI principles <cit.>.

Key components of this audit include:


    
  * Data flow mapping: Tracing the journey of data through organizational systems, identifying potential vulnerabilities and ethical pressure points.
    
  * Policy evaluation: Assessing the alignment of existing policies with HDI principles, paying particular attention to areas such as consent mechanisms, data minimization, and purpose limitation.
    
  * Technological infrastructure assessment: Evaluating current technological capabilities against the requirements of HDI implementation, including privacy-enhancing technologies and explainable AI systems.
    
  * Stakeholder impact analysis: Examining how current data practices affect various stakeholders, including employees, customers, and broader communities.


This comprehensive audit serves as the foundation for a detailed gap analysis, highlighting areas where current practices fall short of HDI standards and identifying opportunities for improvement.



  §.§.§ Stakeholder Engagement and Participatory Design


Concurrent with the audit process, organizations must initiate a robust stakeholder engagement program. This goes beyond traditional consultation models, embracing a participatory design approach that actively involves stakeholders in shaping HDI implementation strategies.

Key stakeholder engagement activities include:


    
  * Multi-stakeholder workshops: Facilitating interactive sessions that bring together diverse perspectives, including data subjects, employees, regulatory bodies, and civil society organizations.
    
  * Ethical deliberation forums: Creating spaces for in-depth discussions on the ethical implications of data practices, encouraging stakeholders to grapple with complex trade-offs and scenarios.
    
  * User experience research: Conducting detailed studies to understand how different stakeholder groups interact with and perceive data systems, informing the design of more human-centric interfaces and processes.


This participatory approach not only enriches the implementation strategy, but also fosters a sense of shared ownership and commitment to HDI principles across the stakeholder ecosystem.



  §.§.§ Strategic Alignment and Roadmap Development


Building on the insights from the audit and stakeholder engagement, organizations must align their strategic objectives with HDI principles. This process involves:


    
  * Vision articulation: Crafting a clear, compelling vision for how HDI will transform the organization's approach to data management and utilization.
    
  * Goal setting: Establishing specific, measurable, achievable, relevant, and time-bound (SMART) goals for HDI implementation.
    
  * Resource allocation: Identifying and securing the necessary resources, including budget, personnel, and technological infrastructure, to support HDI implementation.
    
  * Phased implementation planning: Developing a detailed, phased plan for rolling out HDI initiatives, prioritizing high-impact areas while managing organizational change effectively.


This strategic alignment process ensures that HDI implementation is not treated as a peripheral initiative but is integrated into the core of organizational strategy and operations <cit.>.



 §.§ Phase 2: Policy Development and Technological Integration

The second phase of HDI implementation marks a critical transition from conceptual understanding to practical application. This phase focuses on translating the HDI principles into tangible organizational policies and technological solutions. It represents a pivotal moment where abstract ethical considerations are transformed into concrete operational practices.

Policy development in this context goes beyond mere regulatory compliance, with the aim of integrating HDI principles into the fabric of organizational decision making. Simultaneously, technological integration involves not just the adoption of new tools, but a fundamental reimagining of data systems to support human-centric interactions.

This phase demands a delicate balance between ethical imperatives and operational realities, requiring unprecedented collaboration across diverse organizational functions. It requires a flexible approach that can adapt to the rapidly evolving digital landscape while maintaining a steadfast commitment to HDI principles <cit.>.

Success in this phase sets the foundation for a transformative approach to data management, positioning organizations at the forefront of ethical, human-centered data practices in the digital age.



  §.§.§ Comprehensive Policy Reformation


The second phase focuses on translating HDI principles into concrete policies and procedures. This process goes beyond simply updating existing policies, often requiring a fundamental reimagining of the organization's approach to data governance.

Key aspects of policy development include:


    
  * Ethical framework establishment: Creating a clear, actionable ethical framework that guides all data-related decision-making within the organization.
    
  * Data lifecycle policies: Developing comprehensive policies that address each stage of the data lifecycle, from collection to disposal, ensuring HDI principles are embedded throughout.
    
  * Consent and control mechanisms: Designing policies that empower individuals with meaningful control over their data, including granular consent options and easy-to-use data management interfaces.
    
  * Accountability structures: Establishing clear lines of responsibility and accountability for HDI implementation, including the creation of new roles or committees dedicated to ethical data oversight.


These policies should be developed through a collaborative process, involving legal experts, ethicists, technologists, and representatives of various organizational departments to ensure comprehensive coverage and practical applicability <cit.>.



  §.§.§ Technological Infrastructure Enhancement


Concurrent with policy development, organizations must undertake a significant enhancement of their technological infrastructure to support HDI principles. This involves:


    
  * Privacy-Enhancing Technologies (PETs): Implementing advanced PETs, such as differential privacy, secure multi-party computation, and homomorphic encryption, to protect individual privacy while maintaining data utility.
    
  * Consent Management Platforms: Developing or adopting sophisticated consent management systems that allow for dynamic, context-aware consent processes.
    
  * Explainable AI Systems: Integrating explainable AI technologies to enhance the transparency and interpretability of algorithmic decision-making processes.
    
  * Data Provenance and Auditing Tools: Implementing robust systems for tracking data lineage and enabling comprehensive audits of data usage and transformations.


The selection and implementation of these technologies should be guided by a thorough assessment of their alignment with HDI principles and their potential impact on stakeholder experiences.



  §.§.§ Organizational Capacity Building


To support the successful implementation of new policies and technologies, organizations must invest heavily in building internal capacity for HDI. This includes:


    
  * Comprehensive Training Programs: Developing and delivering tailored training programs for different organizational roles, covering both technical aspects of HDI implementation and ethical decision-making in data-driven contexts.
    
  * Change Management Initiatives: Implementing structured change management processes to facilitate the cultural and operational shifts required for HDI adoption.
    
  * Skills Development: Investing in upskilling and reskilling programs to ensure staff have the necessary competencies to operate effectively within an HDI framework.
    
  * Cross-functional Collaboration: Fostering environments and processes that encourage collaboration between different organizational functions, breaking down silos that may impede HDI implementation.


These capacity-building efforts are crucial for ensuring that HDI principles are not just understood theoretically but are actively applied in day-to-day operations across the organization <cit.>.



 §.§ Phase 3: Implementation and Evaluation




  §.§.§ Phased Rollout and Iterative Refinement


The implementation phase should follow a carefully structured, phased approach:


    
  * Pilot Programs: Initiating small-scale pilot programs in selected departments or processes to test HDI implementations in controlled environments.
    
  * Feedback Loops: Establishing robust mechanisms for gathering and analyzing feedback from stakeholders involved in pilot programs.
    
  * Iterative Refinement: Using insights from pilot programs to refine policies, technologies, and processes before broader rollout.
    
  * Scaled Implementation: Gradually expanding HDI implementations across the organization, maintaining flexibility to adapt to emerging challenges and opportunities.


This phased approach allows organizations to learn from their first experiences and make the necessary adjustments, increasing the likelihood of a successful adoption throughout the organization.



  §.§.§ Continuous Monitoring and Adaptive Governance


As the principles of HDI become more widely adopted, organizations need to set up thorough monitoring and governance frameworks:


    
  * Real-time Monitoring Systems: Implementing technologies and processes for continuous monitoring of HDI compliance and effectiveness across all data operations.
    
  * Stakeholder Feedback Mechanisms: Creating multiple channels for ongoing stakeholder input, including regular surveys, focus groups, and open feedback platforms.
    
  * Ethical Review Boards: Establishing cross-functional ethical review boards to assess complex cases and guide policy evolution.
    
  * Adaptive Governance Frameworks: Developing flexible governance structures that can evolve in response to new ethical challenges, technological advancements, and regulatory changes.


These monitoring and governance mechanisms ensure that HDI implementation remains responsive to changing contexts and emerging ethical considerations.



  §.§.§ Impact Assessment and Reporting


Regular, comprehensive impact assessments are crucial to understanding the effectiveness of HDI implementations and identifying areas for improvement.


    
  * Multidimensional Impact Metrics: Developing a set of metrics that capture the multifaceted impacts of HDI implementation, including ethical, social, economic, and operational dimensions.
    
  * Longitudinal Studies: Conducting long-term studies to track the evolving impacts of HDI practices on various stakeholder groups and organizational outcomes.
    
  * Transparent Reporting: Producing regular, detailed reports on HDI implementation progress, challenges, and impacts, making these accessible to all stakeholders.
    
  * External Audits: Engaging independent third parties to conduct periodic audits of HDI practices, ensuring objectivity and credibility in evaluations.


These assessment and reporting practices not only drive continuous improvement but also contribute to building trust and accountability with stakeholders.



 §.§ Phase 4: Adaptation and Evolution




  §.§.§ Horizon Scanning and Proactive Adaptation


To remain effective in a rapidly evolving digital landscape, organizations must develop robust capabilities to anticipate and respond to emerging trends and challenges.


    
  * Technology Foresight: Establishing dedicated teams or processes for monitoring and analyzing emerging technologies and their potential ethical implications.
    
  * Regulatory Tracking: Developing systems for staying abreast of evolving data protection regulations and proactively adapting HDI practices to ensure compliance.
    
  * Ethical Foresight: Engaging in ongoing ethical deliberation to anticipate future ethical challenges arising from new data uses or societal changes.
    
  * Collaborative Innovation: Participating in industry collaborations, academic partnerships, and multi-stakeholder initiatives to collectively address emerging HDI challenges.


This proactive stance enables organizations to stay ahead of the curve, adapting their HDI practices to address new challenges before they become critical issues.



  §.§.§ Continuous Learning and Knowledge Management


Building a robust knowledge management system around HDI practices is essential for long-term success:


    
  * Case Study Development: Documenting and analyzing specific cases of HDI implementation, including successes, challenges, and lessons learned.
    
  * Best Practice Repositories: Creating and maintaining accessible repositories of HDI best practices, updated regularly with new insights and experiences.
    
  * Cross-organizational Learning: Facilitating knowledge sharing across different parts of the organization to ensure consistent application of HDI principles.
    
  * External Knowledge Exchange: Engaging in knowledge exchange with other organizations, industry bodies, and academic institutions to broaden the collective understanding of effective HDI practices.


These learning and knowledge management practices ensure that the organization continuously builds on its HDI capabilities, fostering a culture of ongoing improvement and innovation.



  §.§.§ Long-term Strategic Integration


Finally, organizations must work towards fully integrating HDI principles into their long-term strategic planning:


    
  * Strategic Visioning: Regularly revisiting and refining the organization's long-term vision for ethical data management, ensuring it remains aligned with evolving HDI principles and societal expectations.
    
  * Innovation Alignment: Ensuring that all data-driven innovation initiatives are evaluated through an HDI lens from their inception, making ethical considerations a fundamental part of the innovation process.
    
  * Organizational Culture Evolution: Continuously working to embed HDI principles into the core values and culture of the organization, making ethical data practices an intrinsic part of organizational identity.
    
  * Stakeholder Relationship Evolution: Developing more collaborative, trust-based relationships with stakeholders, positioning the organization as a leader in ethical data practices.


This long-term strategic integration ensures that HDI is not treated as a separate initiative, but becomes an integral part of how the organization operates and creates value in the digital age.

By following this comprehensive and in-depth roadmap, organizations can embark on a transformative journey towards ethical, human-centered data practices. This approach not only ensures compliance with evolving regulations but positions organizations at the forefront of responsible innovation, building trust and creating sustainable value in an increasingly data-driven world.




§ ENHANCING HDI CULTURE

For an organization to truly embody HDI principles, a deep-rooted understanding of these concepts among all members is vital. Regular training sessions should be conducted to educate employees about the ethics of data use, responsible data handling, and the potential risks associated with improper data management. Furthermore, it's important to extend educational efforts to stakeholders, including users and affected communities, thereby increasing their data literacy. This empowers them to participate actively and effectively in data governance processes, ensuring their voices are heard and considered in decision-making.

However, cultivating an HDI-centric culture within an organization is not a one-time achievement but a continuous process. It requires the establishment of mechanisms for ongoing monitoring, feedback, and iterative improvements. Such processes are crucial for maintaining the effectiveness of HDI practices amidst the rapidly changing dynamics of data use and technology <cit.>

By committing to constant monitoring and adaptation, an organization showcases its dedication to continuously enhancing its HDI practices. This proactive approach not only fosters trust among stakeholders but also positions the organization as a frontrunner in responsible and ethical data management. As the digital environment evolves, staying ahead in implementing robust HDI frameworks becomes a testament to an organization's commitment to ethical data use and governance.

Training and education form the cornerstone of implementing HDI frameworks effectively. This comprehensive approach is key to building a culture that values ethical data use, adheres to HDI principles, and equips both employees and users with the necessary skills and knowledge to navigate the complexities of contemporary data ecosystems. The outlined strategies underscore the importance of developing targeted training and education programs that are integral to fostering an environment where HDI principles are not just understood but actively practiced.



 §.§ Employee Training Programs

Equip employees with a deep understanding of HDI principles, including data ethics, responsible data handling, and awareness of biases. To further empower individuals as active stakeholders in the data ecosystem, the HDI Framework calls for proactive user education initiatives <cit.>. This training should extend beyond theoretical knowledge, emphasizing practical skills for implementing HDI in their daily responsibilities.

Strategies:


  * 

Curriculum Development: Design a curriculum that covers essential topics such as data privacy, consent protocols, algorithmic transparency, and bias mitigation. This curriculum should be adaptable, allowing for updates as new ethical challenges and technological developments arise <cit.>
  * 

Interactive Learning: Employ interactive learning techniques, such as case studies, simulations, and role-playing exercises. These methods encourage active engagement and help employees internalize HDI concepts by applying them to real-world scenarios<cit.>.
  * 

Cross-Functional Training: Implement cross-functional training sessions to foster interdisciplinary understanding and collaboration<cit.>. Exposing employees to interdisciplinary perspectives and diverse challenges across different organizational departments can foster a comprehensive approach to implementing HDI principles.





 §.§ Stakeholder Education

Empower users and affected communities by enhancing their understanding of data rights and the implications of data practices. Education programs should aim to demystify data operations and foster informed participation in data ecosystems.

Strategies:


  * 

Accessible Materials: Develop educational materials that are accessible to non-experts, including infographics, videos, and simple guides. These resources should explain users' data rights, how their data is used, and ways they can control their personal information<cit.>.
  * 

Community Engagement: Organize workshops and forums in collaboration with community organizations. These events can serve as platforms for dialogue, where users can express concerns, ask questions, and provide feedback on data practices
  * 

Feedback Mechanisms: Establish mechanisms for users to provide ongoing feedback on data practices and education materials. This feedback loop is essential for ensuring that education programs remain relevant and responsive to users' needs.





 §.§ Continuous Learning and Adaptation

 Foster an environment of continuous learning and adaptation to address the dynamic nature of data ethics and technology. Training and education programs should evolve in response to new challenges and insights.

Strategies:


  * 

Regular Program Updates: Update training and education programs regularly to incorporate the latest research findings, ethical debates, and technological advancements. This ensures that all stakeholders remain informed about current best practices.<cit.>
  * 

 Professional Development Opportunities: Offer professional development opportunities for employees to deepen their expertise in data ethics and HDI-related fields. This could include sponsoring attendance at conferences, online courses, and certification programs.<cit.>
  * 

Evaluation and Feedback: Implement robust evaluation processes to assess the effectiveness of training and education programs. Use feedback from participants to make iterative improvements, ensuring that learning objectives are met.<cit.>





 §.§ Cross-functional Collaboration:

Implementing HDI principles effectively requires a collaborative effort that spans various departments and expertise within an organization. Cross-functional collaboration is essential for addressing the multifaceted challenges of ethical data management and for ensuring that HDI practices are consistently applied across all organizational processes.
Strategies for fostering cross-functional collaboration in HDI implementation include:


    
  * Interdisciplinary Teams: Form teams that bring together professionals from diverse backgrounds such as data science, legal, ethics, user experience, and business operations. This diversity ensures that HDI considerations are viewed from multiple perspectives, leading to more comprehensive and nuanced solutions <cit.>.
    
  * Collaborative Workshops: Organize regular workshops where team members from different departments can share insights, discuss challenges, and collaboratively develop solutions to HDI-related issues. These sessions can help break down silos and foster a shared understanding of HDI principles <cit.>.
    
  * Shared Responsibility: Establish HDI as a shared responsibility across all departments, rather than confining it to a single team or unit. This approach ensures that HDI principles are integrated into all aspects of the organization's operations <cit.>.
    
  * Cross-departmental Training: Develop training programs that bring together employees from different departments to learn about HDI principles and practices. This not only enhances knowledge sharing but also builds relationships across functional boundaries <cit.>.
    
  * Collaborative Tools and Platforms: Implement collaborative tools and platforms that facilitate communication and knowledge sharing about HDI practices across different teams and departments <cit.>.
    
  * Rotational Programs: Consider implementing rotational programs where employees can spend time in different departments to gain a holistic understanding of how HDI principles apply across various organizational functions <cit.>.


By fostering cross-functional collaboration, organizations can ensure a more holistic and effective implementation of HDI principles. This collaborative approach not only enhances the quality of HDI practices but also promotes a culture of shared responsibility for ethical data management across the entire organization.


The development and implementation of training and education programs are foundational to the success of HDI frameworks. By prioritizing comprehensive, accessible, and continuously updated training for both employees and users, organizations can cultivate a culture of responsible data stewardship. This not only aligns with ethical imperatives but also enhances trust and cooperation across the data ecosystem, paving the way for a future where data practices are transparent, fair, and human-centered.




§ DATA LIFECYCLE MANAGEMENT

Data Lifecycle Management (DLM) is a crucial component of the HDI Framework, ensuring that ethical considerations and human-centric practices are applied at every stage of data handling. This comprehensive approach addresses the entire journey of data from its creation to its eventual disposal, aligning each phase with HDI principles.



 §.§ Collection

The data collection phase must prioritize ethical practices and user empowerment. Organizations should implement transparent mechanisms for obtaining informed consent, clearly communicating the purpose and scope of data collection. Privacy-by-design principles should be embedded in the collection process, ensuring that only necessary data is gathered <cit.>.



 §.§ Processing

Data processing should adhere to the principles of fairness, transparency, and accountability. Implementing explainable AI techniques can help make algorithmic decision-making processes more transparent and understandable to users <cit.>. Regular audits of data processing systems should be conducted to identify and mitigate potential biases.



 §.§ Storage

Secure and responsible data storage is paramount. Organizations must implement robust security measures to protect data from unauthorized access or breaches. Data minimization principles should be applied, storing only essential information for the required duration. Clear policies on data retention and deletion should be established and communicated to users <cit.>.



 §.§ Utilization

The utilization of data should be guided by ethical considerations and respect for individual rights. Organizations should implement mechanisms that allow users to understand how their data is being used and provide options for controlling its use. Ethical review processes must be in place for new data utilization projects, especially those involving sensitive information or potentially high impact decisions <cit.>.



 §.§ Sharing

Data sharing practices must prioritize user consent and control. Clear policies should govern the circumstances under which data can be shared, with whom, and for what purposes. Implement granular consent mechanisms that allow users to selectively share their data and revoke access as needed <cit.>.



 §.§ Disposal

The final stage of the data lifecycle, disposal, must be handled with care. Organizations should establish clear protocols for securely deleting data when it is no longer needed or when users request its removal. This process should be verifiable and in compliance with relevant data protection regulations <cit.>.

By meticulously managing each stage of the data lifecycle in alignment with HDI principles, organizations can foster a more ethical, transparent, and user-centric approach to data handling. This comprehensive lifecycle management not only enhances compliance with regulatory requirements but also builds trust with users and stakeholders, ultimately contributing to a more responsible and sustainable data ecosystem.



§ MONITORING AND ADAPTATION

The implementation of HDI principles within an organization is an ongoing, dynamic process that requires constant vigilance and flexibility to adapt to the ever-changing landscape of regulations, technological advancements, and societal expectations. To maintain and enhance the effectiveness of HDI practices, organizations must establish robust mechanisms for continuous monitoring, regular reassessment, and agile adaptation of practices <cit.>.



 §.§ Continuous Assessment


Organizations should implement structured processes for regularly reviewing their data handling and interaction policies. This involves not only examining the internal mechanisms of data management and governance but also actively seeking feedback from a wide range of stakeholders, including employees, users, and external communities affected by the organization's data practices <cit.>. Techniques such as employee surveys, user satisfaction assessments, and the establishment of a dedicated data ethics committee can provide invaluable insights into the efficacy of existing practices and highlight areas requiring improvement or adjustment.



 §.§ Feedback Mechanisms


Establishing comprehensive feedback channels that enable stakeholders to actively report concerns, provide suggestions, and share their perspectives is crucial. This continuous feedback loop is essential for driving iterative improvements to the organization's data governance policies, technological solutions, and overall HDI implementation <cit.>.



 §.§ Adaptive Governance


As the data landscape evolves, so too must the policies and practices that govern it. Organizations should develop flexible governance structures that can evolve with changing needs, regulatory requirements, and technological capabilities. This may involve regular policy reviews, updates to data management protocols, and adjustments to decision-making processes <cit.>.



 §.§ Emerging Technology Integration


Stay informed about emerging trends, technologies, and regulatory developments that could impact ethical data practices. Continuously research and analyze the implications of new advancements in areas such as privacy-enhancing technologies, consent management platforms, and explainable AI systems. Adapt the HDI framework to proactively incorporate these advancements, ensuring that the organization remains at the forefront of ethical data interaction <cit.>.



 §.§ Performance Metrics


Develop quantifiable metrics to gauge the performance of HDI initiatives. These might include user consent rates, the volume of data access or modification requests, incident reports relating to data security breaches, and evaluations of algorithmic bias. Incorporating these metrics into visual dashboards can facilitate broader organizational awareness and engagement with HDI performance indicators, bridging the gap between technical and non-technical stakeholders <cit.>.



 §.§ External Audits


Carry out regular independent evaluations of HDI practices. Third-party audits can offer an unbiased review of the organization's compliance with HDI principles, unveil possible blind spots, and suggest new perspectives for enhancement <cit.>. By cultivating a culture of continuous improvement and adaptability, organizations can maintain and develop their HDI practices over time. This strategy guarantees that HDI practices stay relevant and responsive to the requirements and issues of all stakeholders, thereby preserving trust and confidence in the organization's data practices and contributing to a more ethical and responsible data-driven society <cit.>. 

The successful implementation of HDI relies on an organization's capacity to remain agile, responsive, and dedicated to ethical data practices amid rapid technological and societal changes. By adopting this dynamic approach to monitoring and adaptation, organizations can not only achieve current standards for responsible data management but also predict and tackle future challenges in the constantly evolving digital environment. 

The process of monitoring and adjusting HDI practices is fundamentally linked to the intricate and changing nature of the data ecosystem. As organizations strive to uphold effective HDI frameworks, they must also navigate a landscape marked by complex data systems, emerging technologies, and evolving societal expectations. This balance between ongoing monitoring and the need to address complex, dynamic factors highlights the importance of a comprehensive, forward-looking approach to HDI implementation. 

The following section examines these complex and dynamic factors, discussing how organizations can effectively manage the complexities of data systems, foster an HDI-centric organizational culture, and stay responsive to developing trends and context-specific challenges. By dealing with these factors alongside robust monitoring and adaptation practices, organizations can ensure that their HDI frameworks stay both effective and relevant in the face of a continuously changing digital environment <cit.>.



 §.§ Complex and dynamic considerations


Implementing HDI frameworks within organizations involves navigating a complex landscape of ethical and practical considerations. These considerations are critical for developing data practices that not only comply with technical standards but also align with ethical imperatives and societal expectations.



  §.§.§ Complexity

Data systems within contemporary organizations exhibit remarkable complexity<cit.>. These systems frequently evolve incrementally, encompass diverse stakeholder interests, and operate across varied technological platforms. Implementing HDI principles in such intricate environments requires a nuanced and strategic approach.

A modular strategy, which focuses on applying HDI principles to specific and well-defined aspects of the data lifecycle, can be effective. Successes in these targeted areas can build organizational confidence and momentum, facilitating a gradual expansion of the HDI principles in broader operational domains. This phased strategy helps manage the risk of becoming overwhelmed by the wide range of possible modifications required for complete HDI integration<cit.>.



  §.§.§ Envolving Technologies

The data landscape is constantly changing, shaped by rapid technological advancements, evolving regulatory environments, and changing societal expectations<cit.>. Organizations must remain vigilant and agile in adapting their HDI frameworks to these dynamic circumstances. 

This necessitates a continuous process of monitoring, evaluation, and refinement, whereby organizations regularly assess the efficacy of their HDI practices and make adjustments as needed.Mechanisms for gathering user feedback, tracking emerging best practices, and proactively scanning for regulatory changes must be embedded within the organization's operational structures. 

By maintaining a pulse on the evolving HDI landscape, organizations can ensure that their data practices remain relevant, effective, and aligned with the best interests of their stakeholders. 



  §.§.§ Regulatory Landscape:

The regulatory environment surrounding data protection and privacy is in constant flux, with new laws and regulations being introduced or updated regularly. Organizations must stay informed of these changes and adapt their HDI practices accordingly. 

This requires a proactive approach to compliance, involving regular legal reviews and the ability to quickly implement necessary changes to data handling processes. The challenge lies not only in meeting current regulatory requirements but also in anticipating future legislative developments <cit.>.



  §.§.§ Societal Landscape:

Public attitudes towards data privacy and ethical data use are evolving rapidly, influenced by high-profile data breaches, privacy scandals, and increased awareness of the value and vulnerability of personal data. Organizations must be attuned to these shifting societal expectations and be prepared to adjust their data practices to maintain public trust. This may involve going beyond simple regulatory compliance to embrace more stringent ethical standards that align with public sentiment <cit.>.



  §.§.§ Ethical Dilemmas:

As data-driven technologies become more sophisticated, organizations increasingly face complex ethical dilemmas that may not have clear-cut solutions. These could range from questions about the appropriate use of AI in decision-making processes to concerns about the potential misuse of predictive analytics. Addressing these dilemmas requires a robust ethical framework and decision-making process that can navigate the nuanced terrain of data ethics <cit.>.



  §.§.§ Cross-border Data Flows:

In an increasingly globalized digital economy, organizations often need to transfer data across national borders. This presents unique challenges in terms of complying with varying (and sometimes conflicting) data protection laws in different jurisdictions. Implementing HDI principles in this context requires a nuanced understanding of international data protection regulations and the ability to adapt data handling practices to meet diverse legal requirements <cit.>.



  §.§.§ Balancing Innovation and Ethics:

Organizations face the ongoing challenge of balancing the drive for innovation with the need for ethical data practices. While data-driven innovation can lead to significant advancements and competitive advantages, it must be pursued in a way that respects individual privacy and upholds ethical standards. This balance requires careful consideration of the potential impacts of new data uses and technologies, and the development of governance structures that promote responsible innovation <cit.>.



  §.§.§ Adaptive Strategies: 

Given the rapidly evolving nature of the data landscape, organizations need to develop adaptive strategies that can respond quickly to new challenges and opportunities. This involves creating flexible governance structures, fostering a culture of continuous learning and improvement, and developing the capacity to pivot data practices in response to emerging ethical concerns or technological advancements. Adaptive strategies should also include regular reviews and updates of HDI frameworks to ensure they remain relevant and effective <cit.>.

The complex and dynamic considerations inherent in implementing HDI frameworks inevitably give rise to a host of challenges that organizations must address. As we have seen, factors such as the intricacy of data systems, the need for adaptive organizational cultures, and the rapidly evolving technological landscape create a multifaceted environment in which HDI principles must be operationalized. 

These complexities naturally lead to specific challenges that require innovative solutions. In the following section, we will explore these challenges in detail and propose practical, research-backed solutions to overcome them. By examining issues such as technical integration, ethical decision-making, user empowerment, and regulatory compliance, we can provide a comprehensive roadmap for organizations seeking to navigate the intricate terrain of HDI implementation. 

This approach not only addresses the complexities discussed earlier but also offers tangible strategies for overcoming obstacles, thereby enabling organizations to more effectively embed HDI principles into their core operations <cit.>.



§ HDI FRAMEWORK CHALLENGES AND SOLUTIONS

Implementing a HDI framework within an organization encompasses a myriad of challenges, ranging from technical complexities to ethical dilemmas. Addressing these challenges effectively requires a multifaceted approach that incorporates both innovative solutions and a commitment to ethical principles. This section outlines key challenges associated with HDI implementation and proposes viable solutions, grounded in scholarly research and best practices.



 §.§ Technical Complexities

Challenge: Integrating HDI principles into existing data management infrastructure. Organizations often have well-established data pipelines, analytical tools, and storage systems in place.

Solution: Adopt a modular and iterative approach to HDI integration, focusing on specific, well-defined aspects of the data lifecycle.This allows organizations to build momentum and confidence through early successes, before gradually expanding HDI principles across broader operational domains.




 §.§ Data Complexity and Integration

Challenge: Modern organizations operate within highly complex data ecosystems that have evolved over time. These systems often feature diverse data sources, varying data formats, and legacy systems that may not easily support new HDI principles.

Solution: Adopting a modular, phased approach to implementing HDI principles can help organizations manage the complexity. Initiating small-scale, pilot projects to integrate HDI principles in specific areas can provide valuable insights and allow for gradual expansion across the organization. Leveraging data integration tools and adopting standards for data interoperability can also facilitate smoother integration of HDI principles.




 §.§ Ethical Decision-Making and Bias Mitigation

Challenge: Ensuring ethical decision-making and mitigating biases in algorithms and datasets represent significant challenges, as these biases can lead to discriminatory outcomes and undermine trust in data system.

Solution: Implementing robust ethical review processes and adopting transparent decision-making frameworks can enhance accountability. Furthermore, integrating bias detection and correction methodologies, such as algorithmic audits and inclusive dataset design, can actively mitigate biases, promoting fairness and non-discrimination in data practices.




 §.§ User Empowerment and Participation

Challenge: Truly empowering users within the HDI framework requires moving beyond nominal consent mechanisms to enable meaningful user control over their data.

Solution: Developing and deploying consent management platforms that offer users granular control over their data, alongside clear, accessible information about data use practices, can enhance user empowerment. Additionally, creating channels for user feedback and participation in decision-making processes reinforces user agency and ensures that HDI practices align with user expectations and values.




 §.§ Organizational Culture and Leadership

Challenge: Cultivating an organizational culture that genuinely prioritizes HDI values, such as transparency, accountability, and ethical data use, requires significant shifts in norms and practices.

Solution: Leadership commitment to HDI principles is crucial. This can be demonstrated through resource allocation, setting clear ethical guidelines, and recognizing employees who champion responsible data stewardship. Fostering a culture of continuous learning and ethical reflection, where data ethics are integrated into everyday practices, can embed HDI values deeply within the organizational fabric.




 §.§ Regulatory Compliance and Evolving Standards

Challenge: The dynamic nature of data regulation and the emergence of new technological capabilities necessitate ongoing vigilance and adaptability to ensure compliance and ethical integrity.

Solution: Establishing dedicated teams to monitor regulatory developments and technological advancements can help organizations stay ahead of compliance requirements and ethical considerations. Engaging in industry consortia and standard-setting bodies can also provide insights into emerging trends and best practices, facilitating proactive adaptation of HDI practices.




 §.§ Cost Implications

Challenge: The implementation of an HDI framework can entail significant costs, including investments in new technologies, training programs, and potentially restructuring data governance practices. Organizations may face financial constraints that limit their ability to fully adopt HDI principles.

Solution: Strategic planning and prioritization can help mitigate cost concerns. Organizations should assess the most critical areas for HDI integration based on risk, impact, and strategic value, focusing initial investments where they can achieve the most significant benefits. Leveraging open-source technologies and platforms for data management and ethics training can also reduce costs. Additionally, incremental implementation allows for the spreading of costs over time, making the process more financially manageable.




 §.§ Data Literacy Gaps

Challenge: A fundamental barrier to HDI implementation is the varying levels of data literacy across an organization, from executive leadership to operational staff. Insufficient understanding of data principles can hinder effective decision-making and limit the capacity to engage with HDI tools and policies meaningfully.

Solution: Comprehensive data literacy programs are essential for bridging these gaps. Such programs should cater to different roles within the organization, providing tailored training that ranges from basic data awareness to advanced analytical skills. Furthermore, creating a culture that values continuous learning and curiosity about data can encourage self-driven improvement in data literacy across the organization.




 §.§ Balancing Innovation with Responsibility

Challenge: Organizations striving to innovate with data often encounter tensions between pushing the boundaries of what is technologically possible and adhering to ethical data practices. This balance is crucial to maintaining trust and ensuring long-term sustainability.

Solution: Establishing clear ethical guidelines that define responsible innovation practices is vital. These guidelines should encourage creativity and exploration while setting firm boundaries to prevent unethical data use. Regular ethical reviews of new projects and technologies can ensure that innovation aligns with HDI principles. Engaging with external ethics experts and stakeholders can also provide diverse perspectives that enrich the organization's approach to balancing innovation with responsibility.

In conclusion, the HDI framework presents a comprehensive model for a future driven by data, where the needs, concerns, and values of humans are central to the design and deployment of data-driven technologies. Implementing an HDI framework is a complex undertaking that requires addressing technical, ethical, and organizational challenges. To navigate these challenges effectively, organizations should adopt a strategic, phased integration approach, commit to ethical principles, and foster a culture of transparency and accountability. By doing so, they can enhance their data practices and contribute to building a more equitable and trustworthy digital ecosystem.

Addressing the cost implications, data literacy gaps, and the need to balance innovation with responsibility is crucial for the successful integration of HDI principles into organizational practices. By adopting strategic, inclusive, and ethical approaches, organizations can navigate these challenges effectively. This not only enhances their capability to implement HDI frameworks but also strengthens their position as leaders in responsible data management and innovation.




§ CONCLUSIONS AND FUTURE WORKS


The Human-Data Interaction (HDI) Framework presented in this article marks a fundamental paradigm shift in the conceptualization and implementation of data management and utilization practices within organizations. This framework represents a radical departure from traditional data-centric approaches, which often prioritize technological capabilities and organizational efficiency over human concerns. Instead, HDI places human values, rights, and agency at the forefront of data practices, recognizing that data systems exist within a complex socio-technical ecosystem where ethical considerations are paramount.

By centering human needs and experiences, HDI offers a comprehensive model for navigating the intricate ethical landscape of our increasingly data-driven future. This approach acknowledges the profound impact that data practices have on individual lives, societal structures, and the distribution of power in the digital age. It seeks to rebalance the relationship between individuals and data systems, moving away from viewing people as mere data subjects to recognizing them as active agents with inherent rights and dignity.

The HDI framework addresses the multifaceted challenges arising from the pervasive use of data in modern society, including issues of privacy, consent, algorithmic bias, and the opacity of data-driven decision-making processes. It provides a structured approach for organizations to ethically manage the entire data lifecycle, from collection and processing to analysis and application. This holistic perspective enables organizations to anticipate and mitigate potential ethical pitfalls, fostering trust and accountability in their data practices.

Moreover, the HDI framework serves as a bridge between technological innovation and ethical responsibility. It challenges the notion that these two aspects are inherently at odds, instead proposing that true innovation in the data sphere must be grounded in respect for human values. By doing so, it paves the way for a future where technological advancements and ethical considerations evolve in tandem, mutually reinforcing each other to create data ecosystems that are not only powerful and efficient but also just, transparent, and human-centric.

In essence, the HDI framework represents a critical evolution in our understanding of the role of data in society. It offers a roadmap for organizations to navigate the complex ethical terrain of the digital age, ensuring that as we harness the transformative power of data, we do so in a manner that upholds human dignity, promotes societal well-being, and cultivates a more equitable digital future for all.



 §.§ Key Takeaways and Implications

The Human-Data Interaction (HDI) Framework presented in this article offers a transformative approach to data management and utilization, with significant implications for organizations, individuals, and society. As we synthesize the core elements of this framework, several key takeaways emerge, each providing actionable insights for the practical implementation of ethical data systems.

These takeaways encapsulate the essence of the HDI framework's contribution to data ethics and provide a roadmap for responsible data management in the digital age. They challenge us to rethink the relationship between humans and data, proposing a paradigm where ethical considerations are foundational to data systems.

The implications of these insights extend beyond data management, influencing organizational culture, technological innovation, and societal values. They set the stage for future research and development in HDI, highlighting areas needing further investigation and pointing towards emerging trends in ethical data practices.
In the subsequent points, we will delve into each of these key insights, analyzing their importance, obstacles, and possibilities for fostering a more fair, open, and human-focused data environment.


    
  * Ethical Foundations: At the core of the HDI framework are foundational principles including agency, legibility, equity, accountability, privacy, and stakeholder engagement. These principles serve as the ethical bedrock for all data practices, ensuring that human dignity and rights are upheld in increasingly data-centric environments. The implementation of these principles represents a significant departure from traditional data management approaches, necessitating a fundamental reevaluation of how organizations collect, process, and utilize data. This ethical foundation challenges organizations to move beyond compliance-driven approaches to data management and instead embrace a proactive stance that anticipates and addresses ethical concerns throughout the data lifecycle.
    
  * Holistic Approach: HDI necessitates a multidisciplinary approach, integrating insights from technology, law, ethics, and user experience design. This holistic perspective enables organizations to address the multifaceted challenges of ethical data management comprehensively. By breaking down silos between different domains of expertise, HDI fosters innovative solutions that are both technically robust and ethically sound. This integration of diverse perspectives is crucial for developing data practices that can withstand scrutiny from various stakeholders and adapt to evolving societal expectations.
    
  * Structural Implementation: The framework provides practical guidance for embedding HDI principles into organizational structures, including data governance frameworks, technological infrastructure, and human-centric practices. This structural approach ensures that HDI is not merely aspirational but operationally integrated. The challenge lies in adapting existing organizational structures and processes to accommodate these new ethical imperatives. This may involve creating new roles, such as ethics officers or data stewards, redesigning data workflows to incorporate ethical checkpoints, and developing new metrics for measuring the ethical performance of data systems.
    
  * Adaptive Strategies: Given the rapidly evolving nature of technology and data practices, the HDI framework emphasizes the importance of continuous monitoring, feedback, and adaptation. This dynamic approach ensures that HDI practices remain relevant and effective in the face of emerging challenges and opportunities. Organizations must develop the capability to anticipate and respond to changes in the technological, regulatory, and societal landscape. This might involve establishing dedicated teams for horizon scanning, regularly reviewing and updating ethical guidelines, and fostering a culture of ethical reflection and debate within the organization.
    
  * Cultural Transformation: Implementing HDI requires a significant shift in organizational culture, emphasizing transparency, accountability, and ethical data stewardship. This cultural change is crucial for the long-term success and sustainability of HDI practices. It involves not only training and education but also a fundamental realignment of organizational values and priorities. Leaders must model ethical behavior, incentivize ethical decision-making, and create an environment where employees feel empowered to raise ethical concerns. This cultural transformation extends beyond the organization to its relationships with customers, partners, and the broader community, fostering a new paradigm of trust and collaboration in the data ecosystem.


 By embracing these key takeaways and implications, organizations can begin to realize the full potential of the HDI framework, creating data practices that are not only innovative and efficient but also ethical, transparent, and deeply respectful of human values and rights.



 §.§ Future Research Directions


As the field of HDI continues to evolve, several key areas warrant further investigation:


    
  * Adaptive Ethical Frameworks:
    
        
  * Develop dynamic ethical guidelines that can evolve with technological advancements.
        
  * Investigate methods for real-time ethical assessment of data practices.
        
  * Explore the use of AI in monitoring and enforcing ethical data standards.
    
    
  * Advanced Privacy-Enhancing Technologies (PETs):
    
        
  * Research next-generation encryption methods for data protection.
        
  * Develop improved techniques for anonymization and pseudonymization that balance privacy with data utility.
        
  * Investigate the potential of homomorphic encryption in enabling secure data analysis on encrypted data.
    
    
  * Explainable AI (XAI) for HDI:
    
        
  * Advance methods for making complex AI decision-making processes interpretable to non-expert users.
        
  * Develop user-friendly interfaces for explaining algorithmic decisions.
        
  * Investigate the impact of XAI on user trust and engagement with data systems.
    
    
  * Cross-Cultural HDI Implementation:
    
        
  * Conduct comparative studies on HDI implementation across different cultural contexts.
        
  * Develop culturally adaptive HDI frameworks that respect local norms while upholding universal ethical principles.
        
  * Investigate the impact of cultural differences on user perceptions of data privacy and agency.
    
    
  * Quantifying HDI Impact:
    
        
  * Develop metrics and methodologies for measuring the effectiveness of HDI implementations.
        
  * Conduct longitudinal studies on the impact of HDI on organizational performance and user trust.
        
  * Investigate the economic implications of HDI adoption for businesses and industries.
    
    
  * HDI in Emerging Technologies:
    
        
  * Explore the application of HDI principles in emerging fields such as the Internet of Things (IoT), edge computing, and quantum computing.
        
  * Investigate the implications of HDI for human-AI collaboration and coexistence.
        
  * Develop HDI frameworks for immersive technologies like augmented and virtual reality.
    
    
  * HDI and Data Ecosystems:
    
        
  * Research methods for implementing HDI principles in complex, interconnected data ecosystems.
        
  * Investigate the challenges and opportunities of HDI in decentralized data systems, including blockchain technologies.
        
  * Develop strategies for maintaining HDI principles in data sharing and collaborative data environments.
    
    
  * Regulatory Frameworks and HDI:
    
        
  * Analyze the interplay between HDI principles and evolving data protection regulations.
        
  * Develop policy recommendations for incorporating HDI principles into legal frameworks.
        
  * Investigate the potential for HDI to serve as a bridge between technological innovation and regulatory compliance.
    
    
  * HDI Education and Literacy:
    
        
  * Develop comprehensive educational programs to enhance data literacy among the general public.
        
  * Create curricula for training future data professionals in HDI principles and practices.
        
  * Investigate effective methods for cultivating an HDI-centric organizational culture.
    
    
  * HDI in Crisis and Pandemic Scenarios:
    
        
  * Explore the application of HDI principles in emergency data sharing situations.
        
  * Develop frameworks for balancing public health needs with individual data rights.
        
  * Investigate the long-term implications of emergency data practices on public trust and HDI implementation.
    


In conclusion, the HDI Framework offers a robust foundation for ethical, transparent, and human-centric data practices. As we navigate the complexities of our data-driven future, continued research and development in these areas will be crucial. By advancing our understanding and implementation of HDI, we can work towards a future where technological progress and human values are not in conflict, but in harmony. The journey towards fully realizing HDI principles within organizational practices is complex and challenging, but the potential benefits—a more ethical, transparent, and equitable data ecosystem that empowers individuals and upholds fundamental human values—are profound and far-reaching.



§ ADDITIONAL INFORMATIONL



 §.§ Funding

This work has received partial support from the National Project granted by the Ministry of Science, Innovation and Universities, Spain, under Grant PID2022-140974OB-I00, and from the Regional Government (JCCM) and the European Regional Development Funds (ERDF) through the INTECRA Project under Grant SBPLY/21/180501/000056.



 §.§ Declaration of interests

The authors state that they have no financial interests or personal relationships that could have influenced the work presented in this article.



 §.§ Author contributions

All authors contributed to the preparation and analysis of the article, as well as the drafting of the manuscript.


unsrt
  





























 Paper Title: A State-of-the-Art Review of Computational Models for Analyzing Longitudinal Wearable Sensor Data in Healthcare.















































A State-of-the-Art Review of Computational Models for Analyzing Longitudinal Wearable Sensor Data in Healthcare
    Paula Lago 
 paula.lago@concordia.ca 
 Department of Electrical and Computer Engineering, PERFORM Center 
Concordia University, Montreal, QC, Canada
    
========================================================================================================================================================

§ ABSTRACT 
Wearable devices are increasingly used as tools for biomedical research, as the continuous stream of behavioral and physiological data they collect can provide insights about our health in everyday contexts. Long-term tracking, defined in the timescale of months of year, can provide insights of patterns and changes as indicators of health changes. These insights can make medicine and healthcare more predictive, preventive, personalized, and participative (The 4P's). However, the challenges in modeling, understanding and processing longitudinal data are a significant barrier to their adoption in research studies and clinical settings. 
In this paper, we review and discuss three models used to make sense of longitudinal data: routines, rhythms and stability metrics. We present the challenges associated with the processing and analysis of longitudinal wearable sensor data, with a special focus on how to handle the different temporal dynamics at various granularities. 


 We then discuss current limitations and identify directions for future work. This review  is essential to the advancement of computational modeling and analysis of longitudinal sensor data for pervasive healthcare.



§ INTRODUCTION


Modern phones and wearable sensors have emerged as indispensable tools for capturing continuous, longitudinal, real-world behavioral and physiological data for health and health research <cit.>. The unprecedented depth of information, combined with advances in machine learning, opens the door to further our understanding of the relationship between health and behavior, to automatically assess and predict health status, and to create personalized interventions to improve health outcomes. 






One practical challenge to realize the potential of research using such sensors is how to make sense of the large amounts of data collected over long periods. As stated by Xu. et.al. <cit.>, the information captured by these sensors, "while information-rich, is also information-vague", requiring thoughtful analysis to obtain valuable insights. 
A large body of research has focused on finding features in the data that correlate with a specific health condition, for example, COVID-19 <cit.>, epilepsy <cit.>, depression <cit.>, schizophrenia <cit.>, anxiety <cit.>, or mild cognitive disease <cit.> among others. While the results are encouraging, few standards exist for the analysis. More often than not, the features are heavily dependent on the device used and hard to generalize <cit.>, which makes it difficult to translate research findings into policy or recommendations.

Longitudinal data can support applications such as early warning systems by identifying changes in behavior <cit.>. One critical aspect for these applications is finding typical patterns in the data. These typical patterns can be used to summarize the data, find anomalies, and measure the strength of the "typicality" or "variability" in day-to-day life. Research has shown associations of behavior regularity with physical health <cit.>, and disruptions of daily routine were found to be associated with mental health disorders <cit.>, thus providing a foundation for the need to find objective measures of regularity and disruption using sensor data. For instance, activity patterns measured with passive sensors were found to be associated with Mild cognitive disease <cit.>.
However, the lack of standards to analyze sensor data has made results obtained so far non-generalizable and difficult to replicate <cit.>.
Consequently, there exists a pressing need to develop theories, tools, and techniques that not only model a baseline of typical patterns but also adeptly measure, interpret, and comprehend variations or changes to such baseline.

This work provides a comprehensive review of existing methods to model behavior and predict health using continuous, longitudinal sensor data. Two complementary views have emerged in the literature to address this problem: routines and rhythms. Both views underscore temporal aspects of behavioral and physiological regularity: routines focus on the frequency of events; rhythms emphasize temporal regularity and periodicity. Recently, stability metrics have also been proposed based on routine or rhythms views. 
We highlight their strengths and differences as well as the common open research challenges ahead. 

Other reviews exist on the use of smartphones and wearable sensors in healthcare. A part of these reviews focuses on the application areas and evidence around the use of passive sensing for early diagnosis, management, or prediction of disease <cit.>. Other focus on the technologies accelerating the rise of passive sensing in healthcare. 
Fewer reviews focus on the modeling approaches for longitudinal patterns. Mohr et.al.<cit.> focuses on mental health applications, providing an overarching model of personal sensing, and reviewing the literature on using sensors to detect mental health conditions and related behavioral markers.
The review by Witt et.al.<cit.> describes algorithmic approaches used to generate fitness- and health-related indicators from wearables data, all of which focus on providing a current picture of the user instead of focusing on their long-term patterns. They briefly review behavior change detection techniques which focus on the analysis of longitudinal data. 

Meyer et.al. <cit.> have reviewed the implications of long-term tracking, clearly differentiating it from short-term tracking. Their review focuses on implications for the user collecting the self-track data. They define long-term "in timescales of years". Here, we review existing models that may support the modeling of such data for healthcare applications. 

In this review, we focus on the modeling long-term patterns, their analysis, quantification, and the main issues in making sense of the large amounts of data collected. We first provide a description of the rhythm (Section <ref>) and routine (<ref>) models and their evolution over time, and then provide an overview of common metrics used to quantify the regularity of patterns (Section <ref>). We finalize the review with a discussion of open issues in both models (Section <ref>).
In an effort to unify the clinical and computational perspectives, this review synthesizes the rhythmic and routine-based approaches to long-term pattern modeling, as well as the growing focus on metrics. This review contributes to the biomedical research by providing a comprehensive summary of the current methods to model, analyse and process longitudinal wearable sensor data. 










§ RHYTHMS


A rhythm is a cyclic pattern, repeated at regular temporal intervals. Human behavior and physiology follow biological rhythms, the most common ones are linked to the day/night cycle. These rhythms are governed by an internal pacemaker called the suprachiasmatic nucleus (SCN), an internal structure located in the brain, and are affected by external factors like light, temperature, and social factors, among others <cit.>. Research has found strong links between the strength and alignment of this central clock with environmental and behavioral cycles and our physical and mental well-being. For this reason, the characterization of the various behavioral and physiological rhythms from sensor data is deemed as an important next step in understanding these rhythms in everyday life conditions to design personalized diagnostics and interventions.  

Longitudinal measures of activity and physiology obtained with passive sensing have been used to either model the SCN rhythm or rhythms of processes such as activity, heart rate, or body temperature. While the latter may not strongly correlate with the SCN rhythm, as they are affected by various other factors, their characterization can still provide information about health status. Moreover, the study of multiple rhythms can shed light on how these rhythms and their alignments affect our health. We now explain four rhythm models. 



 §.§ Template curves

One simple way of modeling typical rhythms of the data is to model a "template" curve for the day.  A template curve is created by averaging (or calculating the median value) across all days for each time slot in the day (e.g., every minute, hour). Activity curves <cit.> were proposed to model the probability of each activity happening at each time slot when the measurements are not continuous signals but rather, events. 

These curves can be used for anomaly detection if values deviate more than a certain amount from the template. The main issue is defining the time over which to calculate the template. They have also been used to obtain deviation from the template as a feature for classification purposes. 

While calculating template curves is simple, one of the major drawbacks of curves is the lack of context. A single template curve cannot account for differences due to day-to-day variations or monthly variations. Of course, template curves could be created for each day of the week or every day of the year, as is done with weather data, but it could lose representational power. 



 §.§ Cosine models

Rhythms are most commonly modeled as sinusoidal functions, characterized by multiple parameters. The first parameter is its period. As such, circadian rhythms are those with a 24-hour period ( the average circadian period in humans is 24.2h), while an ultradian (infradian) rhythm is one with a period of less (more) than 24h <cit.>. While 24 hours is usually assumed for the period, due to its relevance for the internal master clock, frequency analysis methods like the Fourier Transform, wavelets, spectral analysis, and periodograms can be used to extract personalized periods from the longitudinal sensor data. This period analysis can help uncover individual variations, as well as various rhythms. 

By modeling rhythms as a sinusoid, its amplitude, MESOR, and Phase describe the rhythm of behavioral or physiological signals. Phase measurements are of particular relevance as they enable the description of the relative timing of physiological rhythms with environmental and behavioral cycles. These metrics are obtained by fitting the sensor data to a cosine wave of a predefined period using the least square method — this is called the COSINOR method <cit.>. Equation <ref> shows a single component COSINOR, other models include additional components to consider, for instance, the effect of light in the rhythm. Interested readers are referred to <cit.> for a complete discussion of circadian rhythm models.   


    Y(t) = M + Acos(2π/τ + ϕ ) + e(t)



While characterizing the rhythms of physiological and behavioral processes is important, the importance of understanding the relation between multiple rhythms should not be disregarded. Afsaneh et.al.<cit.> proposed a framework that combines period finding with COSINOR to independently describe rhythms of various signals and then uses these descriptors to predict depression risk. Runze et.al. <cit.> also models a cosine rhythm for each signal and finds that rhyhtmic models are correlated with productivity. While these are  preliminary results in a specific population, they shows the potential of combining multiple rhythms in the development of digital health services.  



 §.§ Models to predict SCN rhythm

As previously mentioned, one common goal of modeling rhythms through sensor data is to model the SCN rhythm: predict its period, phase, and amplitude. The gold standard to measure the SCN rhythm is by taking multiple samples of melatonin, cortisol, or core body temperature under highly controlled conditions like dim light or a constant routine protocol <cit.>. These procedures are costly and invasive, so passive sensing provides a cost-effective and ecological way to predict it. 

The COSINOR method has been used with actigraphy (accelerometer-based) and physiological data to model the SCN rhythm with relative accuracy in ambulatory settings. To account for influences on the measurement other than the circadian rhythm, various mathematical models of the clock have been proposed <cit.>. 

In most cases, though, passive sensor data is used to predict the onset of melatonin (time of peak), which is a gold standard metric of the SCN phase <cit.>. Deep learning and traditional machine learning techniques involving sensor measurements of activity, skin temperature, heart rate, and other physiological processes have been proposed for this purpose <cit.>. While these methods are promising, other studies have shown that the error could be higher in populations who may benefit the most from precise metrics such as shift workers, jet-lagged people, and people with sleep disorders <cit.>. Novel methods still need to be evaluated with more diverse and specific populations.  

Characterizing the SCN rhythm is pivotal to further our understanding of its role in health outcomes, however, researchers also acknowledge that different processes may exhibit different rhythms which need not have the same characteristics as other markers such as the melatonin onset (DLMO). Their characterization is useful for understanding underlying processes and might further increase our understanding of the role of the alignment of various rhythms with health. The methods to analyze such rhythms are similar to those described above, but models may consider the effects of confounder factors. For instance, to analyze the circadian rhythm of heart rate, Bowman et.al.<cit.> first removes the effects of activity, to uncover the underlying rhythm.



 §.§ Rhythm Models excluding Cosine Regression

Since sensor data may not always follow a cosine waveform, non-parametric metrics have also been used to describe the rhythmic properties of sensor data, particularly activity data. These metrics are explained in Section <ref>. Non-negative matrix factorization (NMF) has also been proposed to discover typical daily rhythms <cit.> and multiday <cit.> rhythms without the need for a regression. A dataset of the daily or multi-day sensor signals is created where each column represents an individual's data, which can be daily, weekly, or monthly data. NMF factorizes this dataset (a matrix) to find a small number of common patterns, such that each individual's data can be represented as a linear combination of the patterns. This method is also used for routine modeling (Section <ref>) by creating a matrix of events. 

















§ ROUTINES


One approach to modeling long-term patterns is through routines. A routine is generally thought of as a sequence or set of actions performed regularly. 

While there is no agreed formal definition of a routine, the term is usually used to describe recurring patterns of discrete events such as daily living activities, work tasks, or visited locations. As such, two main dimensions encompass a routine: co-occurring or sequences of events and their temporal recurrence. In contrast to rhythms that focus on the periodicity of sensor measurements, routines focus on the frequent repetition of events.   

Interest in computationally modeling routines is longstanding. Their computational study first emerged as a way to support everyday tasks: routines enable the prediction of the next event, which helps a system provide effective recommendations, assistance, or information for that event. Understanding routines from sensor data has been studied to analyze daily life routines <cit.>, family routines <cit.>, work-place routines <cit.>, and travel routines <cit.>. In the context of healthcare applications, routines help understand behavioral patterns that can be associated with health outcomes either positively or negatively <cit.>.



 §.§ Models of routine

Routines are usually modeled as repetitive sequences of events. As such, to discover routines from sensor data, it is first required to define and identify events. What makes a sequence "repetitive" is considered from multiple points of view: the frequency of occurrence (frequentist approach), the likelihood of occurrence (bayesian approach), and the periodicity of occurrence.  




  §.§.§ Defining events

In the context of identifying routines from sensor data, an event can be defined as a significant occurrence or activity that can be detected or inferred from the data collected by sensors. It represents a specific action, behavior, or state of the individual being monitored.  An important characteristic of events is their duration, that is, they are not instantaneous points in time, but rather an entity with a start and end time. In general, an event characterizes what is being done, by whom, when, and where. 

Events can vary widely depending on the context and the type of sensor data being analyzed. For example, in the context of daily activities, events could include actions like cooking, walking, driving, sleeping, or working out <cit.> while in the context of travel routines, events correspond to locations visited <cit.>. Commonly, an event is described by a set of sensor readings obtained during a time window which may include changes in sensor readings, such as motion, location, temperature, or other relevant parameters, which correspond to the occurrence of the event and can be used to classify them <cit.>. 

Defining and identifying events can be a difficult task. Event definition depends on the sensor technology used and the probable duration of events. In some cases, events are defined directly by the sensor readings obtained during a window of time. In such cases, the number of events can be explosive, and unsupervised learning techniques are used to group them into a more manageable set (see next subsection). Events can also be identified by a , requiring supervised learning to identify them. The activity recognition literature deals with this problem and has been extensively reviewed elsewhere <cit.>. 







  §.§.§ Routines as highly probable events

Initial concepts of routine were based on daily routine, that is, how the day unfolds. The behavior of a particular day can be expressed as a vector b, describing the event that occurred at each particular time. This vector can be categorical, that is, each item bt identifies the event that occurred at time t as a category; or binary, that is, the item b_jt has the value of 1 if the event j occurred at time t. A matrix of multiple days ℬ = D× b   describes the behavior over some time. 

From this matrix, routines can be extracted. The concept of eigen-behaviors, representing the eigenvectors of this matrix, was used by Eagle et.al.<cit.> to find common behaviors in a day. An eigen-behavior represents an intrinsic behavior pattern, eigenbehaviors, then, can be used to describe the overall routine of an individual as any particular day can be described as a linear combination of eigenbehaviors. While a predefined 24-hour period was assumed in this work, different behavior vectors were found for different situations showing that “eigenbehaviors” describe both typical patterns and variations of them.  

Similar to this approach, topic models, a technique of natural language processing to automatically discover topics in a document collection, were used to find "topics" of behaviors <cit.>. In these models, an event is considered a word, each day is a document and a set of days is a document collection. A topic describes a group of behaviors that typically occur together in a day, and each day can be described as a combination of topics. 

One drawback of the previous methods is that they are based on individual data, suffering potentially from a cold-start problem. Only when there is sufficient data from a person, can we learn their routines and make inferences from their behavior patterns. To overcome this drawback, a collaborative-filtering-based approach was proposed by Xu. et al<cit.>, which combines population data with personal data to find relevant behavioral features to infer depression.  

The idea behind these techniques is that there is a set of behaviors occurring at a particular time within predefined duration bins that occur repeatedly. While a particular day exhibits a different combination of these typical patterns, the emerging patterns describe the routines of a person and can be used to represent any particular day. 

The problem with this view is the lack of sequence representation. Each day is described as a “bag-of-events” with positional information. While this representation captures some of the sequential context, it cannot capture partial orders, such as events happening next to each other regardless of the time. For instance, “eigenbehaviors” can find patterns such as a late wake-up leading to a no-work day, it is harder to find sequences of activities that can happen together at no particular time. For example, a routine might consist of gym and then eating, but it can occur at different times of the day. Or if there is a sequence of office, gym, and then going home, always starting at 5 pm but having different durations for each depending on traffic, day, and others, it might be hard to discover due to the dependence on time and the predefined time bins. 



  §.§.§ Routines as highly probable sequences

Sequential models analyze data in the order of arrival, thus, capturing dependencies among the events. Both probabilistic and frequentist approaches have been used to model routines as sequences.  From a frequentist perspective, routines are events that empirically occur frequently within a given timeframe based on repeated observations, while from a Bayesian perspective, routines involve incorporating prior beliefs or knowledge about the likelihood of events occurring and updating these beliefs with new evidence.

Frequent-sequential pattern mining and association rule mining <cit.> algorithms work under the frequentist assumption to identify events occurring repeatedly together in an event log. The extracted routine is then expressed as rules <cit.>, action maps <cit.>, or as a grammar <cit.>.
Frequent sequences can represent typical sequences of events in the routine of a person but it is harder to model the uncertainty of multiple possible sequences. Another approach was proposed to consider context-based sequences <cit.>, to model frequent sequences that depend on the context such as day of the week or the weather, independently of the frequency of that context. 

From a probabilistic perspective, Bayesian Networks, which include hidden Markov models, have been used to learn and model the sequential dependencies of events <cit.>. These networks model the transition probabilities between events, as well as the correlation between events and sensor readings. A common problem is defining the structure of the network to learn the parameters. Banovic et.al. <cit.> proposed a model based on reinforcement learning to learn the structure. The learned network of events describes the behavior of the person. By considering the context, such as location or time, in the description of the event, variations in behavior are modeled as well. However, the combinatorial explosion of the number of events and the need to manually describe them make it a computationally expensive model. 



   

Both models have been used extensively in research, their main strength being the possibility of inspecting the routine and knowing exactly what is the next event, making them appropriate for intervention design. Their other strength is the possibility of modeling contextual changes. This is done either by adding it to the event definition or as part of the mining processing. In sequential patterns, modeling includes deciding the maximum length of the sequence and whether gaps or different orderings can be included. These decisions may affect the final learned routine. 

Deep learning models have also been used to model longitudinal sensor data. For this, the events are defined by slices of data which can be a day of sensor events. Due to their black-box nature, Deep learning models have mostly been used in tasks to associate behavior with health outcomes. Recently, Xu et.al. <cit.> proposed a 1d Convolutional network to detect depression, which also learns the continuity of behavior. This can help learn the behavioral patterns but with the drawback of not being interpretable.
While deep neural networks such as LSTM and GRU have been used to model other sequential data, their use to model routine from sensor data has been limited.  

As mentioned before, models of routines are based on the definition and identification of events. This includes multiple assumptions about the duration of events, often left as predefined bins or windows, which may not accurately represent the duration of all activities leading to errors, and about the accuracy of identification. The propagation of errors from the identification of events to the modeling of routines can affect the usefulness of the model. While the events have also been defined as sensor event counts or statistics over predefined windows of time, this can lead to a combinatorial explosion in the number of possible states or require manual definition of thresholds to limit the possibilities. This is still a major drawback for routine analysis.











§ QUANTIFYING BEHAVIORAL STABILITY WITH LONGITUDINAL SENSOR DATA



Both routines and rhythms model long-term sensor data, focusing on different characteristics of the patterns (periodicity, frequency, sequential dependencies). Other approaches focus on quantifying the strength of repetitiveness without modeling the specific patterns.
A person with high strength (or stability) can be thought of as having a more regular or strict routine while a person with lower strength will exhibit more variability in their day-to-day life. While these metrics can't describe the patterns, they characterize the repetitiveness of behavior. 
The simplicity of a single metric makes metrics attractive to perform statistical analysis, as a tool for early screening, and because they can be more easily interpretable than other models. For instance, less table and more fragmented rhythms have been associated with depression, higher BMI and smoking <cit.>. 
Research focuses on finding the relation between metrics and physical, mental, or cognitive health.  These metrics can be classified as being uni-modal if they consider a single sensor modality, or multi-modal if they consider multiple sensor modalities at the same time. A summary of metrics is presented in Table <ref>, and they are discussed below. 






 §.§ Rhythm-based metrics

Rhythm-based metrics use a single sensor modality to obtain a measure of the strength of the rhythmicity of behavior, that is, how similar the measurements taken at the same over multiple days are. The interday stability, intraday variability,  and relative amplitude (RA) metrics come from circadian rhythm research, as non-parametric measures of the strength of the circadian rhythm and are most commonly calculated using activity counts from actigraphy devices. They measure the strength, fragmentation, and amplitude of the rhythm, respectively. 

The regularity and stability indexes have been proposed to measure the regularity from a variety of sensor modalities, but still consider only one modality at a time. They focus on measuring the similarity of events happening at the same time over multiple days. 
The regularity index measures how similar measurements taken at the same hour over two different days are, while the stability index compares the normalized cumulative distribution value at each hour over two days and then offers a value for a period by comparing all days to each other. 

These metrics rely on having an aggregated measure, usually hourly, to compare the value across multiple days. While the aggregation could be in smaller segments, it is not clear how the aggregation might affect the result. 



 §.§ Event based metrics

Instead of relying on the measurements directly, event-based metrics rely on the detection of an event —i.e., an activity, a location— and then calculating the regularity of occurrences of this event over some time. For instance, the standard deviation in start/end time or duration of the event is a measure of how variable that event behavior is. The sleep regularity index focuses exclusively on the regularity of sleep schedule by measuring if the individual is asleep at the same times over the period being examined. Entropy measures the extent of repetition of events, regardless of their order, considering X as a random variable representing the activities in a day. For a person with a more stable routine, this random variable will have lower entropy, as the events are generated by a more regular process. Entropy measure is chosen for event-based analysis as the events are a categorical variable. The flexible regularity index <cit.> measures the edit distance between two days. By representing each event with a letter and considering the daily sequence of events as a character sequence, the edit distance calculates 







p2cmp5.5cmc 
Metric     Description     Formula 
 3cSensor value-based metrics

         Intraday Stability <cit.>     A measure of how stable a person’s rhythm is over multiple days, calculated from n samples of activity levels x, aggregated hourly q hour-bins. x̅_̅h̅ represents the average activity during hour h. IS scores range from 0, indicative of a total lack of rhythm, to 1, indicative of perfectly stable rhythm     IS= n ∑_h=1^q (x̅_̅h̅ - x̅)^2/q ∑_i=1^n(x_i-x̅)^2

         Interday Variability <cit.>     A measure of the fragmentation of activity or how activity level shifts between two consecutive time slots calculated from n samples of activity levels x, aggregated hourly q hour-bins. IV scores range of 0 to 2 with higher values indicating higher fragmentation     IV= n ∑_i=2^n (x_i - x_i-1)^2/(n-1) ∑_i=1^n(x_i-x̅)^2

         Rest-activity Amplitude <cit.>     Measures the difference between a person’s daily activity (measured as the mean activity level over the most active 10 consecutive hours (M10)) and nocturnal activity (measured as the mean activity during the least active 5 consecutive hours (L5))    RA = M10 - L5/M10 + L5 

         Regularity Index <cit.>     Compares the uniformity of a person’s schedule, comparing hourly (t) data from two days (a,b). Sensor measurements (x_a^t) are normalized to (-0.5, 0.5)     RI_a,b = ∑^T_t=1x_t^a x_t^b/T

         Stability Index <cit.>     Measures the inverse of the median across a period of multiple days P  of the distances between the normalized cumulative activity distributions (C_b^d_1) on two different days. It characterizes the degree to which behavioral patterns varied across the period. A higher index means higher stability.      
SI_b (P)= 1 - median ( D_b(i,j )|i,j ∈ P , i ≠ j 
 where D_b(d_1,d_2) =  1/M∑_i=1^M |C_b^d_1 - C_b^d_2|

         Circadian Rhythm  <cit.>   Measures the strength with which a user follows a 24-hour rhythm in behaviors by calculating the ratio of the energy that fall into the 24 ± 0.5 h period over the total spectrum energy in the 24 ± 12 h period. Here, psd(x) denotes the power spectral density at frequency bin x.     CR = ∫_2π/23.5^2π/24.5psd(x)dx/∫_2π/12^2π/36psd(x)dx

         

         3cEvent-based metrics

         Standard deviation <cit.>     Measures the within-individual variability in a derived parameter (x) of an event. For example, the start time or duration of the event. A higher value indicates more irregular behavior.     s = √(1/N-1∑_i=1^N (x_i - x)^2) 

         Sleep regularity index <cit.>     The probability of an individual being in the same state (s_i,j = asleep or awake) at any two time points 24 hours apart, averaged over the number of days being studied (M). SRI values of 0 represent a random sleep schedule and a score of 100 represents a periodic sleep schedule. N is the number of time points in a day.     SRI = -100 + 200/M (N - 1)*
   ∑_j=1^M∑_i=1^Nδ (s_i,j,s_i+1,j) 

         Flexible Regularity Index <cit.>    Edit-based distance to measure the difference between two days by counting the number of operations (insert, delete, substitute) to convert one day's events into the other day's events. Higher values indicate more dissimilar days.      

         Entropy <cit.>     Measures the repetition of isolated events (i.e. travel events, activities), where E is the set of all possible events. A value of zero indicates no uncertainty (more stability) and a value of 1 indicates all events are equally possible     H(X) = -∑_x in E p(x)log_2p(x)

          [6pt]
         3cMultimodal metrics

         Entropy Rate of a Sequence <cit.>    Measures the extent to which ordered subsequences of events repeat over time. p_n denotes the joint probability distribution of a subsequence of length n     H(X) = 
 lim_x→∞ - ∑_x_1^n ∈ E^n p_n(x_1^n)log_2p_n(x_1^n)/p_n(x_1^n-1) 

         Multiscale Fuzzy Entropy <cit.>    Measures the complexity of a time series    See reference. 

         

Behavior stability/variability metrics



 


 §.§ Multimodal metrics

Previous metrics measure the stability of behavior considering one measured variable, but they can't measure relations across various behavioral and physiological measures such as sleep, steps, and heart rate. Amon et.al. <cit.> proposed an approach based on Multidimensional Recurrence Quantification Analysis (MdRQA) to provide a metric of the stability of behavior from multiple sensor signals. This method quantifies the interactions between multiple signals, thus offering a more holistic view of the repetition of patterns, which can also be understood as contextualizing the signals and their regularity. Similarly, the entropy rate <cit.> looks at the regularity of multiple events instead of an isolated event by considering the sequence of events. 






 §.§ Metrics summary

Metrics are valuable because they provide a quantifiable marker whose correlation with a health outcome can be more easily measured than the whole description of longitudinal data. Metrics can also provide personalized insights as they quantify the individual's behavior over some time. Commonly, researchers use metrics to evaluate their correlation with health conditions such as depression, schizophrenia, or cognitive decline. The strength of that correlation measures the usefulness of the metric.

As was discussed, most metrics rely on a single signal modality and a given time frame, thereby restricting the analysis. It is common to calculate metrics for all signal modalities separately and to calculate them using different groups of days. For instance, it is not uncommon to calculate the standard deviation across all days, across weekdays and across weekends. A combination of these metrics are usually used as input to a machine learning models to improve the predictive power. Multimodal metrics can provide a more holistic or contextualized view of the stability or variability of the behavior. However, it is possible that a single metric can hide some effects. For example, He-Yueya et.al. <cit.> found differences in the meaning of stability. For some signals, greater stability was desired for better outcomes but, for others, more variability was associated with better outcomes.  An improved quantification of the relation between the variability/stability of different signals could help improve our understanding of the influence of behavior in health. 

















§ DISCUSSION


Having examined three approaches to analyzing longitudinal sensor data, we now move on to discussing some open research challenges. 



 §.§ Choosing a model

The first key open question is how to choose a model for long-term sensor data analysis. A key consideration is the goal of the patterns. Common uses include (1) to find associations with health outcomes, to which metrics have an appeal due to their simplicity; (2) to provide assistance and predictions of the next events to which routines may be more suited; and (3) to find change and deviations, to which possibly all three models may help. 

A combination of models is not uncommon, especially in conjunction with machine learning and deep learning for all three tasks. For this, the following open issues need to be solved as well.  



 §.§ Windows and window length

During the analysis of longitudinal sensor data, an important consideration is the amount of time (window length) to analyze. It is important to recognize two different types of windows, neither more important than the other: one window to aggregate sensor data as features or events and another to analyze the patterns in sensor data. 

The first type of window, the event window, defines the length of time that is used to recognize an event or a feature (such as the number of steps taken or the average heart rate) that will later be used to obtain patterns in the form of routines, rhythm, or (ir)regularity metrics. This window is often defined in the range of seconds, a few minutes, or one hour. Extensive reviews have looked at the impact of the size of this window, for example, in activity recognition <cit.>. Automatic segmentation has also been studied for this problem, where the window size is automatically determined by changes in the data distribution <cit.>. 

The second type of window, the pattern window, considers how historical data is used to analyze the patterns, for instance, 7 days or 1 month. Routines, rhythm features, and metrics might change depending on this length. Very little work has been done in the analysis of the impact of this length or how to define it. Most commonly, multiple window lengths are evaluated to find the one with the most predictive value for the problem at hand <cit.>. In general, shorter window lengths will have a narrow but current view of the person's patterns, while longer windows will have a wider view but it could be influenced by outdated patterns— the weight of history is heavy. 

A lack of standards in the size of windows hinders the comparability of results and efforts for data sharing. It is not uncommon to find datasets that aggregate data in a too coarse way, making it harder for others to evaluate or compare results that were obtained with smaller windows. 



 §.§ Baseline establishment and change analysis

Related to the problem of determining the pattern window length is the issue of change analysis. Ideally, when analyzing longitudinal data, one is interested not only in the patterns found in data but also in identifying if any pattern changes have occurred. It is, after all, changes that may be more correlated with health changes or health outcomes after an intervention <cit.>. 
However, most of the research so far focuses on analyzing the current behavioral patterns and the predictive power of these patterns for the health outcome being studied, and there is less research on how to analyze changes in behavior across windows. 

Routine analysis has been used to identify changes in routine and anomalies by comparing current behavior with the learned models <cit.>. To analyze changes in phase or amplitude of circadian rhythms over time, Bayesian spectral analysis and detrended fluctuation analysis have been used <cit.>. Changes are also detected statistically <cit.>. Some approaches based on distribution changes evaluate whether behavior changed in two different windows of time by comparing the distribution of sensor data of one window to that of the other window <cit.>. The work by Cook et. al <cit.> determines behavior change over two time windows as a probability distance. By defining the average probability that an activity occurs during a time interval (i.e. an hour of the day) over the time window, it defines behavior as a probability distribution for each time interval. It then calculates the distance between two time windows using the KL divergence of the two distributions.  


To quantify the change, <cit.> uses the slope of the data to quantify how values are changing inside a given window, but it does not quantify if the stability/rhythmicity is changing over various windows. Cook et.al. <cit.> use a  permutation-based test to determine the statistical significance of detected changes. 


The problem of detecting and quantifying behavior or rhythm change can also be framed as a change point detection problem. While there are multiple time series change point detection methods <cit.>, these methods usually focus on the raw sensor values, which can be less meaningful for wearable sensor data, as the changes might be too granular or too fast. For instance, a changing heart rate due to exercise. For longitudinal data, the relevant changes span weeks or months of baseline pattern.


Other works frame the problem as an anomaly detection problem <cit.>. Anomaly detection algorithms analyze deviations from current patterns to detect abnormal activities, heart rate, or other abnormal values, but they do not focus on identifying if the baseline patterns are changing. In other words, while they detect a momentary deviation, they assume the overall pattern remains constant. 


Overall longitudinal pattern change analysis remains elusive, due to the difficulties in obtaining enough data for the analysis, the difficulties in defining time windows for the analysis, and the difficulties related to the evaluation of any such approaches.  




 §.§ Model Evaluation

Evaluating how accurately the routine, rhythm, or behavioral stability of a person is, is undoubtedly a difficult task. Unlike other tasks, such as supervised classification of activities, there is no ground truth data that can be used to compare the model. 
Routines have been evaluated by letting the user review the output and agree/disagree on it. However, this is still a subjective evaluation. For rhythms, comparison with rhythm parameters of the melatonin or cortisol provides an error measure, but as was discussed, this can be misleading, as different processes may have different rhythms.  

Recently, the evaluation focus has shifted to predictive power, that is, how well the patterns or metrics can detect or predict a health outcome (hospitalization, schizophrenia, depression, etc). These evaluations are important, as they focus on the clinical importance instead of computational metrics, but they are also elusive as they depend on the population in the sample, and have proven difficult to generalize. Moreover, they are difficult to translate to other uses of longitudinal pattern analysis, such as providing assistance with a task or identifying abnormal patterns. 



 §.§ Data Heterogeneity and Data Quality

Related to the evaluation and generalization of results is the data heterogeneity issue. Different platforms, vendors, form factors, users, and software libraries result in significant differences in data <cit.>. These differences can result in very different pattern estimations, whether in routines or rhythms.  While metrics are developed and hand-crafted to find associations with health outcomes, their values can differ significantly due to the high heterogeneity, making it hard to generalize across different domains (sensing platforms, population groups, time) <cit.>. 

For this reason, deep learning methods have been proposed as a way to find more stable features in the sensor data to classify and predict health outcomes that could have behavioral symptoms mainly with multi-task learning to learn patterns in the different domains <cit.>. However, the generalization issue persists. 

A second issue for data quality is the inevitability of missing data <cit.>. When data is being collected in natural settings, missing data may come due to battery issues, non-wearing events, or due to platform restrictions <cit.>. 

Proposals to deal with missing data include interpolation techniques and discarding data that contains missing data over a threshold percentage <cit.>. There are no standards set for this and the correct management of missing data is still an open research issue. 



 §.§ Context

Another issue in the current analysis methods is the lack of context in most of them. Several contextual factors can affect routine, rhythms, and the different metrics, including weekends, holidays, weather, or family visits. Not considering this can result in biased metrics, misunderstanding of the patterns, or a lack of insights into what causes changes in stability.  
Some models of routine consider contextual variations as a way to explain routine deviations and differentiate them from changes in behavioral patterns <cit.>. Some metrics, such as the regularity index, consider the day-to-day similarity, which may consider differences due to day-of-week changes, but not for other contextual factors. As was mentioned, this is also considered in the calculation of other metrics, such as the standard deviation, by filtering the set of days used as input. However, efficient computation techniques to deal with multiple contextual variables, as well as their possibly combinatorial explosion, are less often considered. 

Understanding the association with context might be critical for designing interventions for health, and for providing more holistic insights, therefore, including context in the modeling and analysis of behavioral and physiological signals is critical.  



 §.§ Limitations of this study

This study does not adhere to the methodology of a systematic literature review, and consequently, may be subject to a degree of bias in the selection of papers. The myriad of terms and techniques employed in the field posed a significant challenge to conducting a systematic review. Instead, the primary focus was directed towards identifying and establishing commonalities and distinctions among the various models, as well as their respective applications. The study aims to encompass a wide array of literature to furnish a comprehensive overview of the models pertaining to long-term patterns. 

 







§ CONCLUSION



In this state-of-the-art review, we explored computational models for making sense of longitudinal wearable sensor data for healthcare applications. Our examination of rhythms and routines revealed their distinct roles in capturing temporal patterns.  Additionally, we provide an overview of critical challenges related to data heterogeneity, quality, and temporal granularity for the analysis. 

Both routines and rhythms are used to model typical recurring patterns. While routines focus on the definition of an event, rhythms focus more on periodic patterns on the signal, overcoming the need for specific event recognition. 
Both models describe the data and can be used for predictions (of the next event in the case of routines or the next peak in the case of rhythms). Nevertheless, the evaluation of their accuracy and the results remains challenging as models are often highly dependent on the sensors used.  As a way to mitigate this, researchers have focused on developing metrics focusing on the strength of the routine or how typical any particular day can be.

Several open research challenges exist including choosing the right model for data analysis, determining window lengths for analysis, and dealing with data heterogeneity and quality.  There is a need for context in analysis as a significant gap in current research pertains to the exploration of the interrelation between behavioral changes and physiological alterations. Future research should explore the integration of wearable data with other health-related sources to create a holistic view of individual well-being.



plain




 Paper Title: Algorithm-Assisted Decision Making and Racial Disparities in Housing: A Study of the Allegheny Housing Assessment Tool.



Self-Supervised Models in Automatic Whispered Speech Recognition 
    Aref Farhadipour
Department of Computational Linguistics 

University of Zurich

Zurich, Switzerland.

aref.farhadipour@uzh.ch
Homa Asadi
Department of Linguistics

University of Isfahan

Isfahan, Iran.

h.asadi@fgn.ui.ac.ir
Volker Dellwo
Department of Computational Linguistics

University of Zurich

Zurich, Switzerland.

volker.dellwo@uzh.ch
    August 5, 2024
============================================================================================================================================================================================================================================================================================================================================================






plain
plain



















§ ABSTRACT  
 The demand for housing assistance across the United States far exceeds the supply, leaving housing providers the task of prioritizing clients for receipt of this limited resource.  To be eligible for federal funding, local homelessness systems are required to implement assessment tools as part of their prioritization processes. The Vulnerability Index Service Prioritization Decision Assistance Tool (VI-SPDAT) is the most commonly used assessment tool nationwide.  Recent studies have criticized the VI-SPDAT as exhibiting racial bias <cit.>, which may lead to unwarranted racial disparities in housing provision. Such criticisms have led certain jurisdictions to develop alternative tools. Using data from one such prioritization tool, called the Allegheny Housing Assessment (AHA), we use descriptive and quantitative analysis to assess whether the replacement of the VI-SPDAT with the AHA impacts racial disparities in housing allocation.  We find that the VI-SPDAT tended to assign higher risk scores to white clients and lower risk scores to Black clients, and that white clients were served at a higher rates pre-AHA deployment.  While post-deployment service decisions became better aligned with the AHA score, and the distribution of AHA scores is similar across racial groups, we do not find evidence of a corresponding decrease in disparities in service rates.  We attribute the persistent disparity to the use of Alt-AHA, a survey-based tool that is used in cases of low data quality, as well as group differences in eligibility-related factors, such as chronic homelessness and veteran status.    We discuss the implications for housing service systems seeking to reduce racial disparities in their service delivery.





§ INTRODUCTION

On any given night, roughly half a million people in the United States are experiencing homelessness <cit.>.  Black and African American persons are disproportionately represented among the homeless population, with lifetime rates of homelessness estimated to be more than three times higher for non-Hispanic Blacks compared with non-Hispanic whites <cit.>. Some but not all of this disparity is explained by racial differences in income, incarceration rates, and exposure to traumatic events <cit.>.   Providing long-term supportive housing is the single most effective intervention for reducing homelessness <cit.>.  


As a requirement for participating in programs funded by the United States Department of Housing and Urban Development (HUD), local homelessness systems must implement assessment tools that prioritize the most vulnerable homeless clients for supportive services. The most common housing assessment tool is an interview-based instrument called the Vulnerability Index Service Prioritization Decision Assistance Tool (VI-SPDAT) <cit.>. Problematically, recent studies suggest that the VI-SPDAT is poorly validated <cit.> and racially biased <cit.>. Partly in response to such criticism, homelessness systems across the country have been exploring alternative assessment tools, including ones based on predictive risk models.  

We study one such model developed and deployed by the Allegheny County Department of Human Services (ACDHS).  Allegheny County is a medium-sized US county in Pennsylvania that contains the city of Pittsburgh. Allegheny County's Continuum of Care (CoC) receives HUD funding to coordinate and provide homelessness services such as shelters, supportive housing, outreach, engagement and assessment, and prevention strategies. ACDHS is responsible for managing HUD funding and operating the Coordinated Entry system that assesses and refers clients to CoC providers <cit.>. 

In 2020 the county deployed a tool called the Allegheny Housing Assessment (AHA) to “standardize the assessment process and move away from the interview-based VI-SPDAT approach to an automated assessment based on administrative data that can better predict the likelihood of future adverse outcomes” <cit.>. In this paper, we study the transition from VI-SPDAT to the predictive risk model-based risk assessment, AHA, and evaluate the AHA tool's impact on racial disparities in access to supportive housing services. Specifically, we want to understand: 

    
  * How did the prioritization for housing change when the VI-SPDAT was replaced with the AHA? 
    
  * How does AHA impact the racial disparities in housing service enrollment rates? 
    
  * How does the use of the alternative survey assessment tool (Alt-AHA) impact racial disparities? 
  

While there have been extensive evaluations of the performance, ethics, and algorithmic fairness of the AHA models <cit.>, we are among the first to study the impact of its deployment[We are aware of other ongoing work, but none that has been published.].

Using extensive descriptive and quantitative analysis, we find that prior to the deployment of AHA, VI-SPDAT tended to assign higher risk scores to white clients and lower risk scores to Black clients. While there was no disparity in service rates conditioned on VI-SPDAT scores, the net effect was that white clients were served at a higher rate. Following the adoption of AHA, there was an increase in service to Black clients, especially those at medium- and high-risks. However, these increases paralleled those for white clients, and thus did not result in a reduction in racial disparities.  While there is no evidence of racial disparity among clients assessed with AHA only, there is a significant racial disparity in service rates among clients assessed with Alt-AHA.  White clients were about twice as likely as Black clients to be assessed with Alt-AHA, and conditional on being assessed were much more likely to be served.  The differences in Alt-AHA assessment rates is due in large part to differences in data quality, which is significantly lower for white clients.  Disparities in service rates across groups following an Alt-AHA assessment are at least partly attributable to other explanatory factors that affect eligibility, such as chronic homeless, disability, and veteran status. 

Our study contributes to the literature on algorithmic fairness and human-AI interaction in several important ways. First, to our knowledge, it is the first study to investigate how algorithm-assisted decision-making affects housing disparities using real data from a deployed tool.  Second, we directly study algorithm-assisted decision outcomes of an automated risk assessment tool in the presence of a manual assessment option (the Alt-AHA). Lastly, by identifying important differences between the widely used VI-SPDAT and a machine learning–based alternative (AHA), our study has important policy implications for housing providers.




§ BACKGROUND


In response to longstanding housing shortfalls, in 2008 federal rules began incentivizing housing providers to use risk assessment tools to ensure that the most vulnerable clients are prioritized for supportive housing.  To receive federal funding, HUD requires housing providers to use tools that assess a client's housing and service needs and eligibility, and score their risk vulnerability <cit.>. 



In this section we describe the most widely used tool, the VI-SPDAT, and then detail the development and deployment of the AHA tool that has since replaced the VI-SPDAT in ACDHS operations.



 §.§ VI-SPDAT

The VI-SPDAT was developed by merging OrgCode’s SPDAT (Service Prioritization Decision Assistance Tool) Prescreen tool with Community Solutions’ VI (Vulnerability Index) tool.  It consists of a series of survey questions that Coordinated Entry (CE) workers administer at the time of intake to understand the client’s level of risk and need. The tool has been tremendously popular and, by 2015, was being used in at least 39 states and countries <cit.>.  Until 2020, it was also the tool used in Allegheny County.

Recent research has concluded that the VI-SPDAT is poorly validated <cit.> and racially biased <cit.>. Moreover, the assessment questions can be invasive, asking about drug use, risk-taking behavior, and mental health diagnoses.  Focus groups conducted in Allegheny County showed that former clients had concerns about the long and detailed 45 minute interview the tool entailed, and specifically its potential to re-traumatize.  Despite the invasive questions, a 2018 analysis by ACDHS found that the VI-SPDAT was not effective at identifying clients who are at greatest risk of adverse outcomes associated with housing instability and the scores did not significantly predict risk of return to homeless services. In part in response to criticism, by December 2020 the maintainers of VI-SPDAT had announced that they would phase out the tool <cit.>. 



 §.§ The Allegheny Housing Assessment

In 2014, ACDHS issued a request for proposals (RFP) to explore the use of predictive decision support tools in their operations.  The RFP resulted in a research partnership between ACDHS and the Center for Social Data Analytics at the Auckland University of Technology (CSDA).  By 2018, CSDA began working on a machine learning–based risk assessment model that would eventually become the Allegheny Housing Assessment (AHA).

The AHA tool is based on three separate machine learning models, each trained to predict the likelihood of a given adverse event occurring in the 12 months following a housing services assessment: (i) a jail stay; (ii) a Medicaid-funded inpatient hospital stay for a behavioral health event; and (iii) 4 or more Medicaid-funded Emergency Department events.  Unlike the interview-based VI-SPDAT, the AHA relies not on self-reported answers to interview questions, but rather on demographic and county service utilization data stored in the Allegheny County Data Warehouse <cit.>.    Features considered by the models are drawn from child welfare services, county jail, courts, juvenile probation, behavioral and physical health, previous interactions with homeless services, previous interactions with assisted housing, demographics (other than race), neighborhood data from the American Community Survey, and household characteristics. 

Each of the three models produces a risk score between 1-20, and the three scores are then combined into a single AHA score. Figure <ref> below summarizes the combination step. Scores from the three separate models are binned, summed and then converted to deciles to produce a final AHA score between 1-10, with 10 being the highest risk. 



 §.§ The Alt-AHA Tool

In certain cases, new clients may lack sufficient data to run the AHA tool. For this reason, ACDHS complements the AHA with Alt-AHA, an alternative risk assessment based on a subset of VI-SPDAT questions identified by tool developers as being  most predictive of risk.  Alt-AHA was validated on the same outcomes as AHA, and calibrated to ensure that clients scoring 10 on Alt-AHA have a similar risk of adverse outcomes as those scoring 10 on AHA.

To help guide the use of Alt-AHA, each client on a referral receives a “data quality indicator.”  The indicator is 1 if the Data Warehouse has a unique ID for the individual that has existed for at least 90 days and 0 otherwise. 

CE workers are required to use the Alt-AHA when: (i) the quality indicator is less than 1; or (ii) someone moved to Allegheny County within the last 90 days; or (iii)  someone has not consistently been living in Allegheny County; or (iv) someone is fleeing domestic violence and thus may have been unable to utilize services. Alt-AHA is sometimes used outside of these four cases. When CE workers use the Alt-AHA, they must record their reason for doing so.

An AHA score is always produced, irrespective of whether an Alt-AHA is conducted. The final score used in prioritizing clients, hereafter referred to as “posted risk score”, is the greater of the AHA and Alt-AHA scores.  The next section describes how these scores are used as part of the broader intake process.  




 §.§ Housing Intake Process





The full client intake process is illustrated in Figure <ref>. 
Clients can access the Coordinated Entry system for homeless services by contacting Allegheny Link or through referrals by the Office of Community Services (OCS) Field Units. For clients identified as homeless via the Current Living Situation Screening,  CE workers then conduct an assessment using AHA or Alt-AHA (prior to August 2020, VI-SPDAT).

After screening and scoring, CE workers assess clients' eligibility for specific housing programs. This entails screening for chronic homeless status, which can prioritize them for housing <cit.>. The combination of (Alt-)AHA scores and special circumstances (e.g. veteran status, fleeing from domestic abuse, or disability status) dictates program eligibility. The CE coordinator then places eligible clients on waitlists for the appropriate housing types. If the CE cannot reach a client for updates every 45 days from their last contact, and the client is not enrolled in a street outreach program or shelter, they are removed from the waitlist.

Once a provider has an available slot and the capacity to serve and find a client housing, they request a referral from ACDHS off the waitlist. The housing provider then enrolls clients into their programs, offering case management services and housing assistance. Importantly, this does not guarantee housing; clients might not accept the offer due to personal preferences or circumstances such as hospitalization of incarceration. Despite the AHA tool's key role in housing allocation, human factors like maintaining the wait list and locating clients still play a crucial part in the process, influencing when clients are referred. However, CE workers' ability to alter a client's waitlist position is limited by strict eligibility rules. This is in contrast to other algorithm-assisted decision-making settings such as bail or child maltreatment screening, where human decision-makers have considerably more discretion. 



§ RELATED LITERATURE



Algorithm-assisted decision-making is increasingly used in the public sector, including in homelessness prevention.  Predictive models utilize proxy outcomes, administrative data, and target specific sub-populations to efficiently allocate limited resources in crucial settings. Various studies have used these models to predict outcomes such as high-cost users <cit.>, first-time homelessness <cit.>, returns to homelessness <cit.>, shelter entries <cit.>, and chronic homelessness <cit.>. 
Additional research focuses on specific groups, predicting risks such as daily sheltering arrangements for homeless youth <cit.> and their susceptibility to substance abuse <cit.>, as well as assessing homelessness risks among veterans <cit.>. 

The AHA tool differs from these other approaches by focusing on long-term risks associated with homelessness, providing algorithmic scores to prioritize clients on housing waitlists controlled by ACDHS, rather than binary risk assessments.


Recent studies on predictive tools in housing service allocation reveals several significant challenges. The first concern is the accuracy of administrative data in reflecting clients' past circumstances and urgent needs <cit.>. Research by <cit.> involving interviews with policymakers and caseworkers shows mixed feelings about the objectivity and accuracy of the VI-SPDAT. Factors like clients' trust in caseworkers and their privacy preferences impact what they disclose, which in turn affects scoring.  The scoring also depends on the coordinators' ability to prompt questions and gauge the effects of a client's mood and mental status. Thus, the fairness of these tools may suffer from discrepancies in what is intended to be measured versus what is actually measured and assessed in the AI model <cit.>.   <cit.> note that tools like AHA, which rely on administrative rather than self-reported data, address mismeasurement attributable to faulty self-report, but remain susceptible to other biases in data collection.

The second concern focuses is the use of proxy outcomes in prediction, which may not fully capture the subjective experiences of unhoused individuals <cit.>. Research shows that ML systems might de-prioritize crucial values like efficiency and privacy that are essential for protecting homeless individuals <cit.>. Building trust in these systems at provider and bureaucratic levels is crucial for their support, as noted by <cit.>.

Debate around fairness in housing services also persists. <cit.> highlight a trade-off between fairness and efficiency, where imposing fairness constraints can increase re-entry rates.  <cit.> note a division between service providers, who prioritize vulnerability over race, and policymakers, who seek to address racial disparities in housing access. Some researchers advocate using reparative algorithms to rectify historical inequities <cit.>. Our study specifically evaluates racial disparities in the allocation of housing.

ACDHS commissioned an independent equity evaluation <cit.> of the models prior to deployment, which found that the models operated consistently across racial sub-groups.



§ METHODOLOGY





 §.§ Research Questions


In the study, we seek to understand how CE workers interact with the AHA tool and how the AHA tool impacts racial disparity in housing allocation.   More specifically, we ask three Research Questions (RQ's): 


    
  * How did the prioritization for housing change when the VI-SPDAT was replaced with the AHA? 
    
  * How does AHA impact the racial disparities in housing service enrollment rates? 
    
  * How does the use of the alternative survey assessment tool (Alt-AHA) impact racial disparities? 
  

We study changes in service rates.  An assessment is “served” if it results in an enrollment with a Permanent Supportive Housing (PSH), Rapid Rehousing (RR), or Transitional Housing (TH) provider.  This means the client can start receiving comprehensive case management services, such as mental health and substance use treatment, and housing assistance. As discussed in §<ref>, enrollment does not mean they are assigned to or have moved into housing yet. We do not distinguish between housing types, as all three provide services that address housing instability barriers.
This study is not a comprehensive evaluation of the impact of the tool on overall welfare—only on racial disparities. 



 §.§ Descriptive Analysis and Empirical Model

We conduct descriptive analyses to examine high-level trends, and then more rigorously study the research questions using logistic regressions to control for client characteristics.   Table <ref> of the Appendix provides a summary of key variables and how they are denoted throughout the paper. We also discuss the distinction between demographics and housing eligibility characteristics in Appendix <ref>.

To answer RQ1, we use retrospectively generated AHA scores for the pre- and post-deployment periods to investigate changes in service rates.  We conduct the following regression on clients who were assessed only via AHA, focusing on β_3.  


    Served_i,t    = β_0 + β_1 Post_i,t + β_2 Score  + β_3 Score*Post_i,t
        + β_4 Black_i,t + β_5'X + γ_t + δ_t + ϵ_i,t


β_3 captures the change in association between service decisions and AHA scores from pre to post. We control for client race (Black indicator variable), as well as other controls, X, which include age, gender, veteran status, chronic homelessness status, disability status, etc., as shown in Table <ref>. Time-fixed effects γ_t and δ_t represent assessment year and month, respectively, capture yearly and seasonal variations in service provision. We use robust standard errors clustered by client ID to account for multiple assessments per client and for among-client variation.  

We study RQ2 with the following specification, using data only on clients where AHA (not Alt-AHA) was used.   


    Served_i,t    = β_0 + β_1 Post_i,t + β_2 Black_i,t + β_3 Score_i,t
        + β_4 Risk Group_i,t + β_5 Black * Post_i,t
        + β_6 Risk Group * Post_i,t + β_7 Risk Group * Black_i, t
         + β_8 Risk Group * Black * Post_i,t
        + β_9' X + γ_t + δ_t + ϵ_i,t

This model allows for heterogeneity in the association between service decisions, race, and AHA deployment across risk group strata. The risk strata are defined in Table <ref> as low-risk (AHA scores 1-4), medium-risk (scores 5-8), and high-risk (scores 9-10). We present a simplified short-hand equation here and include an expanded equation in Appendix <ref> for clarification. 

Two coefficients of interest, β_5 and β_8, measure changes in service rates. β_5 reflects how service rates change for the low-risk group post-AHA. β_8 shows the additional service rate change for Black clients in the high-risk group, compared to white clients, pre- and post-AHA deployment, relative to the low-risk group. Essentially, it shows the impact of high risk status on racial disparities post-AHA. As a robustness check, we test alternative specifications in Appendix <ref>, treating AHA scores as continuous or binary for high-risk. The results are consistent with the main findings.

For RQ3, we use logistic regression on data from clients assessed with Alt-AHA, using the following specification where the Alt-AHA score, if higher, is used as the posted score for housing prioritization.


    Served_i,t    = β_0 + β_1 Black_i,t + β_2 Posted Score_i,t
        + β_4Use Alt Score + β_5Quality Indicator
        + β_6' X + γ_t + δ_t + ϵ_i,t


 assesses data quality on a 0 to 1 scale. All other covariates X and time fixed effects remain consistent. The coefficient of interest, β_1, evaluates potential racial disparities in service rates for clients prioritized based on Alt-AHA after controlling for other factors.



§ DATA


We extract housing assessment data from the County’s Homelessness Management Information System (HMIS) for assessments conducted between January 1, 2018 and March 29, 2022. This data is linked with enrollment data for clients in PSH, RR, or TH services during the study period. 

We retrospectively (re-)run the AHA tool to produce scores for pre- and post-AHA deployment periods. We also obtain VI-SPDAT scores from the pre-deployment period. 

We exclude assessments with a race other than white or Black/African American, as they represent only 3.2% of the data. We also exclude 36.1% of the assessments associated with families or youth household types, focusing instead on single adult household types. Prioritization of families or youth households is more complicated and depends on a range of additional criteria not captured by our data. Lastly, we exclude clients who are enrolled with programs that do not require AHA scores. The final dataset contains 6,542 assessments, in which 66.1% were from the pre-AHA period. 399 Alt-AHA assessments were conducted, with 370 using the Alt-AHA scores as their posted scores.

Table <ref> summarizes pre- and post-deployment client characteristics.  We see that the majority of single homeless clients are male, and over 90% report a disability. Post deployment, the prevalence of chronically homeless clients rises. When comparing Black and white clients, their assessment scores, ages, and likelihood of being domestic violence survivors remain similar pre- and post-deployment. However, they diverge in key eligibility criteria: white clients exhibit higher rates of chronic homelessness and disability, whereas Black clients are more likely to be veterans.

Before deployment, Black clients have a lower average VI-SPDAT score (7.5) than white clients (8.4) on a 1-16 scale. Post-deployment, Black clients average a slightly higher AHA score (7.1) compared to white clients (6.6) on a 1-10 scale. White clients are twice (24.4%) as likely to be assessed with Alt-AHA compared to Black clients (11.8%), and to have those scores used for prioritization (20.7% vs. 8.0% for Black clients). This significant difference may be linked to consistently higher data quality indicators for Black clients, with scores of 0.96 vs. 0.87 pre-deployment and 0.91 vs. 0.82 post-deployment.

Finally, white clients were served at a higher rate than Black clients before AHA deployment, 17.6% compared to 14.5%.  The disparity persists post-deployment.






§ DESCRIPTIVE ANALYSIS

We now compare risk scores and service rates from VI-SPDAT, AHA, and Alt-AHA to gain high-level insight relating to our research questions.  In Appendix <ref> we provide a brief analysis of concurrent events, particularly the impact of COVID-19 and lockdown orders, surrounding the AHA deployment. While we note some variation in service rates by race, these changes are not statistically significant.



 §.§ Score comparisons




  §.§.§ AHA vs. VI-SPDAT Scores.

We begin by comparing the AHA and VI-SPDAT scores. Because the VI-SPDAT was no longer conducted after the AHA was deployed, this comparison is based on data during the pre-deployment period.  VI-SPDAT scores range from 1-16 where 1 is the lowest and 16 is the highest risk, and AHA scores range from 1-10.  The left panel on Figure <ref> shows that there is a weak positive correlation between the retrospectively generated AHA scores and the unconverted VI-SPDAT scores (Spearmen rank order coefficient = 0.218, P < 0.0001).  

To facilitate a more direct comparison of the scores for the remainder of the analysis, we bin the VI-SPDAT scores into a 1-10 range.  Scores 1-4 are mapped to 1, 13-16 are mapped to 10, and values in-between are mapped in a one-to-one correspondence to 2:9.  This mapping is monotonic, and is chosen to ensure that there are comparable numbers of clients in the lowest and highest score bins for both the AHA and (post-mapping) VI-SPDAT.   Figure <ref> (right) shows a heatmap of scores after mapping. After conversion, the AHA and VI-SPDAT Scores remain significantly weakly positively correlated (Spearman rank order coefficient = 0.217, P<0.000) 





  §.§.§ AHA vs. Alt-AHA Scores.
 
Figure  <ref> compares AHA scores to Alt-AHA scores for clients who were assessed with Alt-AHA in the post-deployment period.  We see that clients whose initial AHA scores were low (1-3), typically received much higher scores when assessed with Alt-AHA. As a business rule, the higher of the two scores will be used as the score (“posted score”) for prioritization. Alt-AHA and AHA scores do not appear correlated (Spearman rank order coefficient =-0.02, P=0.64).



Figure <ref> shows a breakdown of Alt-AHA use by race.  We observe that white clients are more likely than Black clients to be assessed via Alt-AHA (left panel), and also more likely to then have their Alt-AHA scores used (right panel). These differences are at least partly attributed to lower quality indicators for white clients, as shown in Table <ref>. CE workers provide reasons for disagreeing with AHA scores, which we discuss in Appendix <ref>.  There we find that white clients are more often not residing consistently in Allegheny County, have lower data quality indicators, and have lower service utilization than Black clients. 





 §.§ Score Distribution by Race


Figure <ref> shows the (frequency) distribution of pre-deployment VI-SPDAT and AHA scores, and post-deployment AHA and Alt-AHA scores[A cumulative density version of these plots is shown in Figure <ref> of Appendix <ref>.].  We present within-racial-group frequencies instead of counts so that pre-vs-post comparisons are not confounded by variation in the number of clients being assessed over time.



Comparing the pre-deployment plots, we see that VI-SPDAT scores were lower for Black clients than white clients, with white clients being more likely to receive VI-SPDAT scores in the 8-10 range.  Conditional on VI-SPDAT score, Figure <ref> shows that Black and white clients were equally likely to receive housing services.  However, at least in part because VI-SPDAT scores tended to be lower for Black clients, overall service rates were significantly lower for Black clients than white clients pre-AHA (see Figure <ref>). 

Unlike the VI-SPDAT, in both the pre- and post-deployment period, Black clients were approximately equally likely to receive AHA scores in the 8-10 range.   The third panel of Figure <ref> shows the subsample where AHA scores determined prioritization—this combines clients assessed only via AHA and those assessed via Alt-AHA but whose AHA score was higher. Here, Black and white clients exhibit similar score distributions.  To the extent that differences in prioritization score distributions drive disparities in service rates, this analysis indicates that disparities might decrease once AHA is introduced.  

Second, transitioning from VI-SPDAT to AHA, we observe a significant shift in the score distribution, with a larger percentage of clients across both races now classified as higher risk, whereas previously, most fell into the low to medium risk category. Post-AHA deployment, there is an increased representation of Black clients in the medium and high-risk categories, compared to the VI-SPDAT. 

Finally, in the case of clients assessed via Alt-AHA whose Alt-AHA scores were used for prioritization (where the posted scores are identical to their Alt-AHA scores), the distribution shows a slightly higher percentage of Black clients in the lower risk range and a slightly lower percentage in the high risk range compared to white clients. Given the small sample size, the difference between the distributions is unlikely to be statistically significant.

In summary, we observe a notable disparity under VI-SPDAT, with a higher percentage of Black clients categorized as lower risk and white clients as higher risk. Post-deployment, both Black and white clients receive similar scores under AHA, with Black clients showing an increased presence in the medium to high-risk categories compared to their earlier VI-SPDAT scores. White clients tend to receive higher risk scores when assessed by Alt-AHA. 



 §.§ Service Rates by Race


Figure <ref> shows that, pre-AHA, white clients were served at a significantly higher rate than Black clients  (t-statistic = -2.72, P-value=0.007). Post-deployment, the service rates increased for both groups, but white clients continued to be served at a significantly higher rate than Black clients (t-statistic=-2.15,  P-value=0.032). When we break down post-deployment assessments by whether the Alt-AHA scores were used, we find that the two groups are served at similar rate when AHA scores were used. The racial disparity in service rates appears to be driven mainly by the sub-group of assessments where an Alt-AHA was used. For clients assessed via Alt-AHA, on aggregate, there is a statistically significantly difference (t-statistic: -2.72, P-value: 0.007) between Black and white clients, with white clients (30.0%) about twice more likely to be enrolled with a provider than Black clients (15.4%). However, it is important to note that the sample size for Alt-AHA is small. The aggregate comparison does not account for risk scores or other factors affecting client eligibility, such as veteran status and chronic homelessness. Additionally, white clients generally have lower data quality when assessed using Alt-AHA. We will revisit this more rigorously in § <ref>.



Figure <ref> shows service rates conditioned on AHA and VI-SPDAT scores.  We note that in the pre-deployment period, VI-SPDAT scores strongly correlated with the likelihood of enrolling with a provider. In contrast, conditioned on the retroactively calculated AHA risk scores, the service rates appear uncorrelated with AHA scores.  Post-AHA deployment, we observe service rates aligned more closely with AHA scores, with higher scores more likely to be served.  



Furthermore, conditioned on VI-SPDAT scores, we observe no racial disparities in service rates during the pre-deployment period. However, as mentioned previously, this lack of association does not imply an absence of racial disparity, since VI-SPDAT tended to underestimate the risk of Black clients. If we consider the proxy harms that AHA measures, when CE workers assessed risks using VI-SPDAT without access to AHA scores in the pre-deployment period, white clients where served at higher rates than Black clients among those whose AHA risk score was 9 or 10, as shown in the first panel to the left in Figure <ref>. Post-deployment, the introduction of AHA scores appears to have closed this disparity for clients in the high-risk category. We will test this hypothesis more rigorously in § <ref>.
 
When examining the third and fourth panels in Figure <ref>, which depict service rates for AHA and Alt-AHA assessments, respectively, we observe no significant gap in service for assessments conducted via AHA. While there is a significant overall difference between Black and white clients assessed via Alt-AHA, it disappears when conditioned on Alt-AHA scores—likely due to small sample sizes.



 §.§ Conclusions of Descriptive Analysis

Addressing RQ1, prior to AHA deployment, service rates are highly associated with VI-SPDAT scores, and largely uncorrelated with AHA scores. However, post-AHA deployment, service rates begin to correlate strongly with AHA scores, especially for clients assessed exclusively with AHA.

Addressing RQ2, both pre- and post-deployment, white clients were served at a significantly higher rate than Black clients. While there is no overall disparity in service rates in clients assessed by AHA only, service rates for clients assessed by Alt-AHA are higher for white clients than Black clients.  There is no racial disparity conditioned on VI-SPDAT scores pre-AHA. Yet, retrospective AHA scoring identify two key service rate disparities.  (1) VI-SPDAT tended to assign higher risk scores to white clients compared to Black clients.  By contrast,  AHA scores are more comparable across groups.  (2) Under VI-SPDAT, white clients were served at significantly higher rates than Black clients at the same retrospective AHA risk scores, particularly in the high-risk category. The deployment of AHA reduced the service rate disparity among high-risk clients. 

Finally, for RQ3, we find that post-AHA deployment, white clients were more likely to be assessed via Alt-AHA, their Alt-AHA scores were more likely to be higher than their AHA scores compared to Black clients, and they were more likely to be served. However, this may be due to differences in data quality, demographics, and eligibility criteria. 



§ QUANTITATIVE RESULTS

In this section, we further interrogate the findings from the descriptive analysis using logistic regressions to control for additional explanatory variables. 



 §.§ RQ1


Because we retrospectively (re-)calculated both pre- and post-deployment AHA scores, we can examine whether the switch from VI-SPDAT to AHA resulted in service decisions becoming more closely aligned with AHA scores. 

Table <ref> shows the results of testing this hypothesis using regression specification (<ref>).  In the table columns we progressively control for demographics and eligibility criteria, and then also time fixed effects.



Comparing the  coefficient across columns (1) and (2), we see that once eligibility criteria are controlled for, there's no statistically significant association between decisions and the AHA score pre-deployment.  On the other hand, the coefficient of  is consistently statistically significant across columns, indicating that service rates became significantly positively associated AHA scores post-deployment. Across all columns, the coefficients on  consistently show that Black clients are statistically significantly less likely to be served compared to white clients.   




 §.§ RQ2


As discussed in §<ref>, when CE workers relied solely on VI-SPDAT without access to AHA scores, they tended to serve white clients at a higher rate within the AHA score range 9 and above. We further examine the change in service rate by risk groups using the Equation <ref> allowing for heterogeneous treatment effect proposed in § <ref>. 




We find that that the coefficients of  are consistently statistically significant across the specifications, indicating that high risk white clients are more likely to be served post-AHA deployment compared to pre. 

The coefficients of   and  are not statistically significant for either specification that controls for demographics and eligibility criteria.  We therefore do not have sufficient evidence to conclude that the change in service rates post-deployment is different for Black clients than for white clients for any of the risk strata.  
  
     


 §.§ RQ3


As for the use of Alt-AHA, across all models, higher posted scores and quality indicator scores are consistently associated with a higher likelihood of receiving services through Alt-AHA usage. Conversely, the use of Alt-AHA alone does not significantly impact service likelihood. 

When considering the effect of race and posted score (the higher of Alt-AHA or AHA scores that is used for prioritization for this sub-population) in column (1) and (2) and (3) on service rates, Black clients are significantly less likely to receive services through Alt-AHA compared to white clients, even after adjusting for differences in data quality (in (3)). Once demographic, eligibility and time fixed effects are added to the regression, the race coefficient is no longer statistically significant.  However, this is more due to an increase in the standard error of the coefficient estimate than a clear decrease in the magnitude of the coefficient.  The evidence of higher service rates for white clients under Alt-AHA is therefore borderline.  






 §.§ Conclusion of Quantitative Analysis

In summary, our quantitative analysis confirms, as we would anticipate, that post-AHA deployment, service decisions become more closely aligned with AHA scores. Overall, we do not find evidence that the AHA deployment significantly bridges the disparity in service rates between Black and white clients. The post-deployment change in service rates for medium and high risk Black clients is similar to that observed for white clients.  Lastly, while there is borderline evidence that white clients are more likely to be served when Alt-AHA is used, ultimately the sample size underpinning that analysis is too small to draw strong conclusions.



§ LIMITATIONS

Our study, based on administrative data, has some limitations. We have tried to model the use of AHA risk scores as accurately as possible within real decision-making contexts. However, the complexity of the process from initial assessment to the final service enrollment means we must make some assumptions when interpreting our results.

First, it is essential to acknowledge the possibility that the effects identified in our study are in part attributable to changes during the COVID period related to housing policy and client composition that influenced service allocations.  We further discuss the observed effects of the stay-at-home order and other COVID-era impacts in Appendix §<ref>. 

In order to conclude that the effects we identified are attributable to the deployment of AHA rather than other unmeasured factors, we need to assume that there are no “unobservables” that both (i) changed at the same time as AHA was deployed, and (ii) were relied upon by CE workers in their service decisions. It could be problematic for our study design if, for instance, some information gathered during the assessment process but not reflected in the administrative data influenced CE worker decisions.  We were not able to identify alternative observational study designs that would not be susceptible to potential unobserved confounding.



§ DISCUSSION


We present the first evaluation of the impact of replacing the VI-SPDAT with the Allegheny Housing Assessment on housing service outcomes.  Specifically, we examine how CE decisions change post-deployment, and the effects of those changes on racial disparities in service rates.  While earlier studies of AHA considered the predictive performance and group-fairness properties of the tool's constituent models, they relied on pre-deployment data and did not study the tool's influence on decisions.

We find that white clients were significantly more likely than Black clients to receive high VI-SPDAT scores, but were approximately equally likely to receive high AHA scores.  Later analyses do not, however, enable us to conclude that the shift to AHA-based prioritization significantly reduced the racial disparities in service decisions present under the VI-SPDAT policies.  Pre-AHA, white clients were served at higher rates than Black clients, both overall and conditional on  AHA score.  We find that racial disparities largely persisted post-AHA, particularly in the subset of clients assessed with Alt-AHA.  Our analysis indicates that the persistent disparities are at least in part attributable to differences in data quality, and in differences in eligibility criteria across groups. 

This study has several policy implications for CoC providers considering a switch from the VI-SPDAT to predictive models like the AHA tool or other survey-based assessment such as the Alt-AHA. Our results show that predictive models like AHA can reduce or altogether eliminate differences in the prioritization score distributions.  This, however, does not guarantee that overall disparities in service decisions will be reduced as a result.  Whereas we found no significant post-deployment disparity in service rates among clients assessed with AHA, disparities among clients assessed with Alt-AHA were at least as large as those existing pre-deployment.  This finding underscores difficulties with anticipating the impact of a predictive risk model deployment in settings where the model cannot be run due to poor data quality in a substantial number of cases. 

As discussed extensively in the literature review, there is an important concern about whether administrative data can accurately capture clients' past and current needs <cit.>. Efforts to improve data accuracy and enhance cultural competence among service providers are critical for addressing disparities effectively. Policymakers must consider the strengths and weaknesses of risk assessments derived from administrative data and predictive modeling. Effectively integrating AI with CE workers' expertise could be pivotal in reducing racial disparities in housing service provision.

Future studies should further explore how CE workers use Alt-AHA or deviate from the standard AHA procedure to advocate for their clients. These findings would be invaluable in offering a clearer understanding of the additional risk factors captured during the interview process, and shaping the future iteration of the algorithm. This expanded investigation would significantly contribute to refining and improving the decision-making processes in social services.




§ RESEARCHER POSITIONALITY STATEMENT

All authors have previously conducted evaluations of algorithmic tools in either child welfare or housing services deployed by the ACDHS for academic research. The second author has interned part-time at the ACDHS, gaining firsthand experience with their operational processes and tools. The last author led the development of the AHA tool, and the third author provided feedback on the methodology used in model development and validation.  As a result, the research team collectively has a deep understanding of the design, development, validation and deployment process surrounding the AHA tool.  

We recognize that this experience and familiarity also present the potential for conflicts of interest. In conducting the research described in this paper we maintained a firm commitment to academic integrity and transparency.  As per the terms of our data use agreement, ACDHS was provided with the submission version of this manuscript and given the opportunity to review and provide feedback.  The data use agreement does not enable ACDHS to block the publication.  Through this review process we received helpful feedback that helped improve the clarity of the manuscript, led to a better-motivated model specification for RQ1, and also resulted in several robustness checks now presented in the appendix.  

Our collective expertise includes backgrounds in public policy, statistics, and health economics, enabling a multidisciplinary approach to this evaluation. We are motivated by a shared goal of improving the efficacy and fairness of algorithmic tools in social services.  Our work is informed not only through collaborations with government agencies, but also through engagement with stakeholders most likely to be impacted by the technologies.  


















§ IMPACT OF CONCURRENT EVENTS DURING TOOL LAUNCH ON RACIAL DISPARITY


Figure <ref> shows the service rate for each quarter by race over our analysis period. We highlight three events within our analysis period that could affect service rates and racial disparity: the issuance of the "Stay at Home Order" by Pennsylvania and the state's Department of Health on March 23rd, 2020 <cit.>, the lift of the order in Allegheny County on September 24th, 2020 <cit.>, and one year post the start of the order. These dates are denoted by the purple, green, and yellow dashed lines, respectively, on Figure <ref>. In response to the order and HUD COVID-requirements, ACDHS had to swiftly modify their programs, specifically reducing the number of unhoused clients. During the implementation of the order, we observe increasing service rates for both Black and white clients, aligned with ACDHS's change in business practice. After the AHA deployment, followed shortly by the lift of the order, we observe a short period of fall in the white-Black disparity - but the disparity grew again in favor of white clients one year post after the stay-at-home order. However, it's important to note that these differences are unconditioned on risk levels and also lack statistical significance, a point we assess more rigorously in § <ref>.





§ REASONS FOR USING ALT-AHA

In Figure <ref> we show the reasons for why CE workers disagreed with the AHA scores and decided to conduct Alt-AHA instead. We show the counts of the reasons by race and the percentages of the reasons within each race respectively in the top and bottom graphs. We see that CE workers disagreed with AHA scores for white clients mostly because they do not live consistently in Allegheny county, have poor quality indicators and have low service utilization.  CE workers disagreed with AHA scores for Black clients mostly because they do not live consistently in Allegheny county, have poor quality indicators and did not have client ID (MCI) in the system at the time of contact. 






§ SCORE DISTRIBUTION



In Figure <ref>, we show the percentage of assessments above each scores by race. All assessments has a score one or above, therefore we start with 100% for both white and Black clients. The size of the bars diminishes as there are smaller percentages of higher risk scores. We can interpret the graphs as showing the percentage of clients above a score threshold, where the higher the score the more vulnerable and therefore more eligible for services. We see that both pre- and post-AHA deployment, AHA generates more comparable scores for both racial groups. In contrast, at every scores, there are consistently higher percentages of white clients than Black clients according to VI-SPDAT. With similar service rates under VI-SPDAT, this highlights the scoring mechanism as a potential source of disparity as there are always a higher percentage of white clients above every score thresholds. 




§ RESEARCH METHODS


Here we expand the equation in described in § <ref>. low-risk is 1 if a client's AHA score is between 1-4 and 0 otherwise. medium-risk is 1 if a client's AHA score is between 5-8 and 0 otherwise. high-risk is 1 if a client's AHA score is between 9-10 and 0 otherwise. We chose low-risk group as our base group for comparison. Thus, the coefficient of interest here is 7H which shows the additional difference in change in service rate for Black clients in the high-risk group pre and post-AHA deployment compared to white clients compared to such change in the low-risk group. 


    Served_i,t   = β_0 + β_1 Post_i,t + β_2 Black_i,t
          + β_3 Score_i,t + β_4L Low Risk_i,t
          + β_4M Medium Risk_i,t + β_4H High Risk_i,t
          + β_5 (Black × Post_i,t) 
          + β_6L (Low Risk_i,t× Post_i,t) 
          + β_6M (Medium Risk_i,t× Post_i,t) 
          + β_6H (High Risk_i,t× Post_i,t) 
          + β_7L (Low Risk_i,t× Black_i, t) 
          + β_7M (Medium Risk_i,t× Black_i, t) 
          + β_7H (High Risk_i,t× Black_i, t) 
          + β_8L (Low Risk_i,t× Black × Post_i,t) 
          + β_8M (Medium Risk_i,t× Black × Post_i,t) 
          + β_8H (High Risk_i,t× Black × Post_i,t) 
          + β_9 X + β_10 AltAHA + γ_t + δ_t + ϵ_i,t







§ THE EFFECTS OF DEMOGRAPHICS AND ELIGIBILITY CRITERIA



According to the AHA Methodology Update report <cit.>, demographics include only age and gender—race is not considered when generating AHA scores. Therefore, we classify age and gender as demographics, while categorizing all other features, such as chronic homelessness status, disability, veteran status, and domestic violence survivor status, as eligibility criteria based on discussions with ACDHS (see section <ref> for details about the business process).

In Table <ref>, we regress demographics and a combination of demographics and eligibility criteria on AHA scores using the sub-sample for which Alt-AHA was never conducted, indicating sufficient data quality for AHA scores. This allows us to disentangle the effects of demographics alone and demographics combined with eligibility criteria on AHA scores.

Unsurprisingly, demographics alone explain only 1.92% of the variation in scores, as indicated by the adjusted R-squared value, because demographic features are just a small subset of all features used for prediction. However, after including eligibility criteria, we find that race, demographics, and eligibility criteria together explain 10.70% of the variation in scores, suggesting that eligibility criteria are significantly associated with the scores. Despite this, the explanatory power remains relatively low. Thus, while adding eligibility criteria is expected to reduce the size of the coefficients associated with AHA scores due to collinearity, they will also directly impact service rates.





§ ALTERNATIVE SPECIFICATIONS FOR RQ2


In this section, we examine two alternative specification for RQ2 to illustrate the robustness of the results on the heterogeneous effects of AHA scores on service rate. 

The first alternative specification treats AHA Score as a continuous variable without segmenting assessments into different risk categories. In the second alternative specification, we create a binary variable  which equals to 1 if the AHA Score associated with the assessment is higher or equal to 9, and 0 otherwise. This allows us to more directly distinguish assessments with high risk scores from the rest. The results are shown in Table <ref> and Table <ref> respectively. 

Aligned with the result presented in the main text, we find the coefficients associated with the triple interaction terms of  and  to be not statistically significantly. This shows that the relationship between high AHA scores and service rates does not differ significantly by race in the post-intervention period, supporting the robustness of our findings. 







 Paper Title: Does the Source of a Warning Matter? Examining the Effectiveness of Veracity Warning Labels Across Warners.






    School of Information Sciences,

    Data Science and Engineering, The Bredesen Center,

    University of Tennessee Knoxville, Knoxville, TN, USA

    bhorne6@utk.edu




Does the Source of a Warning Matter? Examining the Effectiveness of Veracity Warning Labels Across Warners
    
    Benjamin D. Horne

    August 5, 2024
==========================================================================================================



§ ABSTRACT 
In this study, we conducted an online, between-subjects experiment (N = 2,049) to better understand the impact of warning label sources on information trust and sharing intentions. Across four warners (the social media platform, other social media users, Artificial Intelligence (AI), and fact checkers), we found that all significantly decreased trust in false information relative to control, but warnings from AI were modestly more effective. All warners significantly decreased the sharing intentions of false information, except warnings from other social media users. AI was again the most effective. These results were moderated by prior trust in media and the information itself. Most noteworthy, we found that warning labels from AI were significantly more effective than all other warning labels for participants who reported a low trust in news organizations, while warnings from AI were no more effective than any other warning label for participants who reported a high trust in news organizations.























§ INTRODUCTION

The rise of online disinformation and misinformation has propelled scientists to focus on understanding and mitigating their impact. This nascent, but quickly growing, interdisciplinary field of study has proposed various systems of safeguards for online platforms to use. One particularly common, yet still understudied, solution is the use of soft content moderation methods. Soft content moderation methods are interventions that do not remove content but instead limit the visibility of content. Soft moderation can be done through attaching veracity labels to content <cit.>, quarantining online communities <cit.>, or demonetizing content producers <cit.>. 

In this study, we focus on one particular method: veracity warning labels. We choose to focus on warning labels because their impact and design are poorly understood, despite already being used in multiple real-life systems <cit.>. While studies have characterized user engagement with warning labels on real platforms <cit.> and have experimented with how humans react to automatically generated warning labels <cit.>, critical gaps in our understanding of label design and effectiveness still exist. As previous work from multiple disciplines has demonstrated: trust in false information, its influence on decision making, and its correction are complex <cit.>. Therefore, it is likely that human interaction with warning labels is also complex.

One thing that is well-known in human-information interaction is that heuristics - rules of thumb for making decisions without exhaustively comparing all options - are used when making decisions about the information we trust <cit.>. One common heuristic is trust in the source of information, sometimes called source credibility cues. That is, we often trust information and spend more time reading information that comes from a trusted source <cit.>. This notion has been well-studied in the context of trusting news stories <cit.>, social media posts <cit.>, political messages <cit.>, and health messages <cit.>, but there has been less attention to how the source plays a role in content warning labels. The warner - the one who gives the warning - can vary across labels. For instance, if a warning label says “False Information. Checked by independent fact-checkers", the independent fact-checkers are the warners. If a label is more generic, such as Reddit's quarantined community label, the warner may be assumed to be the platform itself, Reddit. Given how important source cues are in information consumption, it is reasonable to assume that source cues also play some role in warning label consumption. 

However, from the little work that has been done on source cues in warning labels, there is mixed evidence about relative efficacy across sources <cit.>. For example, <cit.> showed that labels from fact checkers were broadly more effective at deterring the sharing of false news stories than warnings from news media, the public, and Artificial Intelligence (AI). Similarly, <cit.> found that warnings from fact-checkers were more trusted than warnings from machines. Yet, other work has found that warning labels from fact checkers, the public, and algorithms were equally effective at reducing the perceived accuracy of false information for politically liberal information consumers <cit.>. 



Notably, across these studies the target metric of “effectiveness” has differed. In some studies the target metric is trust in information - i.e., does an intervention decrease trust in false information? Similarly, some studies use the metric of perceived accuracy - i.e., does an intervention decrease perceived accuracy of false information? In other studies the target metric is the sharing intention of information - i.e. does an intervention decrease the likelihood that false information is shared on a social network? While these metrics are related, they are not necessarily the same. Somewhat paradoxically, perceptions and actions are not always correlated. The clearest example of this phenomenon comes from a 2021 report that found older people shared more misinformation on social media than young people, and yet older people were actually less likely than young people to believe that misinformation <cit.>. Further, the intention to share information is not necessarily captured by one's perceptions of information <cit.>, and there is some evidence that frequent sharers on social media interact differently with warning label interventions than less active social media users <cit.>.

Hence, in this paper we examine the impact of a warning label source on both the trust in and sharing of false information. We ask three questions that combine and extend the prior work on warning label source cues:

RQ1: Does the source of a warning label change trust in false information? 

RQ2: Does the source of a warning label change the sharing intention of false information? 

RQ3: How do differences in media trust, political identity, and cognitive reflection change warner effectiveness?

To this end, we conduct a five condition, between-subjects experiment to examine the impact of the source of a warning label on the information behaviors of Facebook and Twitter users. Specifically, participants in the study scroll through eight social media posts, half of which are false stories, and indicate how much they trust the information in each post and how likely they would be to share the post. Participants are randomly assigned to one of five warning label conditions: (1) all posts have no warning labels (control), (2) false posts have warning labels from other social media users (crowd), (3) false posts have warning labels from the social media platform, (4) false posts have warning labels from fact-checkers, and (5) false posts have warning labels from an AI tool. Examples of posts in each condition can be found in Figure <ref>.

From this experiment, we find that all warning labels significantly decrease trust in false information relative to the control condition, but warnings from AI are modestly more effective overall. AI had the largest effect relative to control, followed by warnings from fact-checkers, the crowd, and the platform. These impacts are moderated by trust in media and the information itself. For instance, participants who distrust media organizations are less affected by warning labels from the social media platform, other social media users, and fact checkers than participants who trust media organizations. Further, participants who distrust news organizations are significantly more impacted by warning labels from AI than all other warning label sources. On the other hand, AI is no more effective than any other warning label source for participants who trust media organizations.  We also find that political leaning did not robustly moderate the effectiveness of warners. While there is some evidence that conservative participants are less affected by warnings from social media platforms and more affected by warnings from AI in terms of effect size, the interactions between political leaning and each warning condition were non-significant. 





Sharing intentions are only weakly correlated with information trust (+0.3 Spearman's ρ). While the majority of warning labels do significantly decrease sharing intentions relative to control, they have less of an impact on sharing intentions than on information trust. In particular, warnings from other social media users have no significant impact on sharing intentions, while labels from AI, fact checkers, and the platform do. Labels from AI and fact checkers have the largest effects on sharing intentions.



  
Broader Perspectives, Ethics, and Competing Interests
Our findings have implications for content label design and information intervention experiment design. In the long-term, we hope this work, and other work in the ICWSM community, helps build better content moderation methods and policies for a safer information ecosystem. We followed best practices for conducting misinformation research <cit.>.  For example, given the experiment used real information, both true and false, we ensured that all participants went through a debriefing, in which information about the experiment, the veracity of headlines, and an explanation of how veracity was determined for each headline was provided. Additional considerations are provided in the Paper Checklist. The authors declare no competing interests.





§ RELATED WORK

This paper draws from two bodies of literature: 1. the quickly growing work on online content labeling and 2. the much broader literature about trust and heuristics. 



  
Online Content Labels
Online content labels are attachments to content - social media posts, news articles, videos - that are intended to add information or context to that content <cit.>. These labels fall into to two broad categories: 1. context labels and 2. veracity labels. Contextual labels provide additional information about the content, often relating to the source of the content. For example, YouTube's source funding label, which labels videos from government funded outlets <cit.>. Veracity labels, the focus of this study, are more direct than context labels. They are labels that warn consumers of a piece of information's correctness or reliability. For example, Facebook and Twitter have labeled content with third-party fact-checks <cit.>.

Of the experimental evidence produced so far, both contextual and veracity labels impact information behaviors <cit.>. In particular, veracity warning labels seem to be generally effective at reducing both belief in and engagement with false information <cit.>. However, as argued by <cit.> and <cit.>, the contexts and settings in which these findings hold deserve further study.

One such setting change is the source of the veracity labels. As briefly discussed in the introduction to this paper, a few recent studies have directly or indirectly studied the relative efficacy of warning label sources <cit.>, each varying in effectiveness metrics and results. The majority of these works provide evidence that trust in the warner does matter to some degree, but the relative efficacy has varied. Notably, the majority of these studies have focused on specific topical contexts: political information and COVID-19 information. In our work, we want to broaden these results by studying warning label source cues across metrics and topics (our headline selection is similar to that done by <cit.>). Further, our experiment adds an additional source not studied in these prior works: the social media platform. We choose to add this source as many warning label experiments use generic warnings, where the source is not always clear. However, if those warnings were displayed on a specific platform, consumers may assume the platform itself is the source of the warning. The work done in this paper seeks to add both clarity to these prior results and add evidence from a more generic topical context.



  
Information Trust and Heuristics
An important component of labeling content is understanding if labels will be accepted and why labels are or are not accepted. The factors that influence label acceptance stem from a much broader literature on information trust and heuristics. From this literature, we know that trust in information can be evaluated along both affective and cognitive dimensions <cit.>. Trust may change across facets of the information being consumed, such as the familiarity of information or the platform information is consumed on. 

Further, heuristics used in information trust decisions have been well-studied. When heuristics are used in decision-making, we replace a complex, difficult-to-answer question with a simpler, easier-to-answer question (a substitution <cit.>). For instance, as discussed in the introduction, one such heuristic is trust in the source information <cit.>. Rather than deeply processing various facets of the information's truth-status, we may ask “do I trust the source of this information?”. There are many other types of mental shortcuts used in information selection and trust decisions, such as utilizing the writing style of information, perceiving social consensus about the information, trusting information because it forms a coherent story, trusting information shared by significant others, and trusting information that aligns with prior beliefs or knowledge <cit.>.

Veracity warning labels add another layer to information trust decisions. While processing the information itself, whether via a shortcut or deep information processing, likely still plays a part in information behaviors when warning labels are attached, those same shortcuts may also play a role in warning label acceptance. In this study, we focus on testing the influence of one particular heuristic on warning label acceptance: trust in the source of the warning label. However, there are other shortcuts that may be used when choosing to comply with a label. For example, one's prior experience with warning labels may influence their future acceptance of warning labels (like algorithm aversion when users see a system make a mistake <cit.>). Or social consensus about a warning label system may impact trust in warning labels produced by that system, much like how social consensus can influence trust in individual claims. In the context of warning labels, these shortcuts have rarely been studied. This work begins to fill this gap in our understanding of mental shortcuts relationship to a warning label's effectiveness.



§ METHODS















 §.§ Experimental Setup

We recruited 2,215 U.S. adults who regularly use Twitter or Facebook on the survey platform Prolific. Specifically, we use Prolific to recruit participants who answered `Facebook' or `Twitter' to the following question: “Which of the following social media sites do you use on a regular basis (at least once a month)?” This question was a part of Prolific's various pre-screening questions to filter study populations. According to Prolific, 23,727 people were eligible to participate in our experiment who regularly used Facebook and 18,874 people were eligible to participate in our experiment who regularly used Twitter. While it is possible that a participant used both Facebook and Twitter at least once a month, we ensure that the two samples are mutually exclusive. We choose to sample from these two sets of social media users instead of a general U.S. population for two reasons: 1. As discussed below, two of our experimental conditions require a platform to be mentioned, and 2. Prior work has shown evidence that labels have different effectiveness across different platforms. In particular, labels on Twitter were found to be more effective than labels on Facebook <cit.>.

Using these two samples of social media users, we deployed a pilot survey for 15 participants (across both Facebook and Twitter populations) to measure if our estimated survey time and payment amount were correct. There were no issues found during the pilot, so the same payment parameters and questions were used in two more batches of survey deployment (one for Facebook users and one for Twitter users). All three batches were ran across the 18th and 19th of May 2023. Participants were paid $3 for survey completion. Across these three batches 166 didn't finish the survey, leaving us with 2,049 total participants for analysis (1,026 from Facebook and 1,023 from Twitter).



All participants in the study scrolled through eight  randomly-selected social media posts, half of which are false stories. For each post, participants indicated how much they trust the information (5-point scale from Trust Completely to Distrust Completely) and how likely they would be to share the information (5-point scale from Extremely Likely to Extremely Unlikely). Participants were also asked an open-ended question to assess reasoning (“Is there a particular reason you trust (or distrust) the information in this post?”). In total, 16 fact-checked posts were randomly selected from recent headlines/social media posts fact-checked by Snopes, PolitiFact, FactCheck.org, or Reuters Fact Check. We only include recently fact checked headlines to ensure the information is fresh and relatively novel <cit.>. The source and image associated with each headline/post were included in the experiment (as shown in Figure <ref>). Each participant only saw a random selection of half of these, balanced between true and false. In total, this gave us 16,392 post-level data points, 8,196 for false posts and 8,196 for true posts. Importantly, the information could come from any topic and was not limited to a particular type of false information (i.e., political, health, etc.). The headlines used can be found in the supplemental material for the paper.

Each participant was randomly assigned to one of five conditions: (1) All posts have no warning labels (control), (2) False posts have warning labels from the social media platform (Facebook or Twitter), (3) False posts have warning labels from other social media users, (4) False posts have warning labels from an AI tool, and (5) False posts have warning labels from fact-checkers. We choose these four warners because they all have been proposed in the literature or used in real life systems <cit.>.

Prior to being randomly assigned to a condition, participants took a pre-survey to capture additional measures and demographics. The demographics captured included age, gender, race, education, and political leaning. Five additional questions were asked to measure cognitive reflection (CRT), empirically validated by <cit.> and <cit.>. Six more questions were asked to capture participants perceptions of the ability, benevolence, and integrity (ABI) of news media organizations and social media platforms (three questions for news media and three questions for social media). These questions were adapted from the model of trustworthiness proposed by <cit.>. A similar set of measures for cognitive reflection and perceived trustworthiness were captured in a prior warning label study by <cit.>. The survey questions can be found in the supplemental material for the paper.

Finally, after the discernment task, participants were asked two factual manipulation checks to ensure the warning labels were seen. Then the participants were debriefed about what they saw and what information was true and false. 






 §.§ Data Sample and Demographics



The sample across both the Facebook and Twitter populations skewed towards being more liberal, younger, and college educated, with the Twitter population skewing slightly more liberal and younger than the Facebook population. Namely, among the Facebook users, 196 identified as very liberal, 324 as liberal, 290 as moderate, 162 as conservative,and 53 as very conservative. Among the Twitter users, 252 identified as very liberal, 336 as liberal, 232 as moderate, 154 as conservative, and 49 as very conservative.

Using the six ABI questions described above, we compressed answers into an ABI score for news media and an ABI score for social media per participant. Specifically, for each question, if a participant strongly disagreed with the prompt, we subtracted 2 points, somewhat disagreed we subtracted 1 point, neither agreed nor disagreed add 0 points, somewhat agreed we added 1 point, and strongly agreed we added 2 points. This creates a score between -6 and 6, where -6 is the strongest distrust and 6 is the strongest trust. According to this score, participants skewed towards distrusting news media organizations and social media platforms. The median score for news media organizations was -1 (μ = -1.45, σ = 2.56) for Facebook users and -1 (μ = -1.48, σ = 2.60) for Twitter users. Social media platforms were distrusted even more, with the median score for social media being -2 (μ = -2.08, σ = 2.38) for Facebook users and -2 (μ = -2.08, σ = 2.48) for Twitter users.

Similar to our compression for the ABI questions, we compressed the CRT questions into one score per participant. Specifically, for each question, if the participant answered correctly, we add 1 point to their CRT score. This process created a score between 0 and 5, where 0 is the lowest cognitive reflection and 5 is the highest. Overall, participants skewed towards high cognitive reflection, with a median CRT score of 4 (μ = 3.41, σ = 1.53) for Facebook users and 4 (μ = 3.56, σ = 1.53) for Twitter users. Distributions of each demographic variable across the two populations can be found in the supplemental material.  Correlations among these variables can be found in Figure <ref>.



 §.§ Data Analysis

Given the hierarchical structure of our data (repeated measures across individuals), our primary analysis is done using mixed effects regression, also known as a multilevel model. Specifically, we fit two models: one to predict trust in false information and another model to predict sharing intention of false information. Each model has two levels, where the first level is the post and the second level is the participant, thereby clustering the repeated measure of trust in posts and sharing of posts by participants. The dependent variable in each model is a  - a number between -1 (distrust or not share) and +1 (trust or share) which maps directly onto the 5-point Likert scale used in the survey for each post. The independent variables in each model include the condition relative to control, the platform group a participant is in, and participant-level demographics. 

In addition to the post-level trust and sharing score used in the multilevel models, we compute a  for trust in and sharing intention of false posts. The participant-level score is a number between -4 and +4, where -4 is complete distrust in (or extremely unlikely to share) all false posts and +4 is complete trust in (or extremely likely to share) all false posts. For example, a person's trust score in false information is computed as follows: for each false post, add 1 if they answered `Trust Completely', add 0.5 if they answered `Somewhat Trust', add 0 if they answered `Neither Trust nor Distrust', subtract 0.5 if they answered `Somewhat Distrust', and subtract 1 if they answered `Completely Distrust'. This score ignores differences across individual posts and instead captures the higher level shifts in participant-level trust and sharing. We do not use this score for modeling, only for further describing the data.



§ RESULTS


In Table <ref>, we show the pairwise differences between each condition group according to Tukey's HSD Test at the post-level  with Sidak-Bonferroni corrections. In Table <ref>, we show the summaries for two multilevel regression models, one for trust in false information and another for the sharing intention of false information. In Figure <ref>, we show the effect sizes at each level (participant and post) relative to control.














































































All warning labels decreased trust in false information As both analysis at the post-level and participant-level indicate, all warning labels, no matter the source or platform population, significantly decreased trust in false information relative to the control condition (p < 0.001 for all four warning label conditions in model 1). These effects, while significant, were relatively small. As shown in Figure <ref>a, the largest effect size across participant-level score distributions was 0.425 between control and AI and the smallest was 0.272 between control and Platform. Similar, but smaller, effect sizes were found at the post-level. At both levels, AI had the largest effect relative to control, followed by warnings from fact-checkers, the crowd, and the platform.

When examining the differences between each warner, we again see that the label from AI was modestly more effective. In particular, when changing the condition reference in model 1, the difference between warnings from the platform and warnings from AI emerged as significantly different (p < 0.05), but with a very small effect size (0.122 at the participant-level).  This result is partially reflected by Tukey's HSD test in Table <ref>, where labels from AI were found to be significantly more effective than labels from both the crowd and the platform (p < 0.05). However, when adjusting the p-values to correct for family-wise error using Sidak-Bonferroni, the differences between AI and the platform and AI and the crowd become non-significant. Hence, we should emphasize that while labels from AI were very modestly more effective in terms of effect size, they were not robustly more effective. All other sources showed no significant differences from each other, only from control.





Warnings from other social media users were not robustly effective at deterring sharing of false information. While all warning labels were significantly effective at decreasing trust in false information, not all were effective in decreasing sharing of false information. According to model 2 in Table <ref>, warnings from the platform, fact checkers, and AI significantly decreased sharing intentions for false information (p < 0.01 for all three labels). However, warnings from the crowd did not significantly shift sharing intention (although they were found to be significant by the Tukey's HSD test at the post-level in Table <ref>  prior to Sidak-Bonferroni corrections). Of the labels that did impact sharing, the effects again were small (Figure <ref>b). Warnings from AI and fact checkers were the most effective in terms of the effect size relative to control; however, no warner was shown to be significantly more effective than any other according to Tukey's HSD test.

Importantly, while the shifts in sharing intention were small, the baseline sharing intention is already quite low. The average participant-level sharing score in the control condition was -3.26, where the lowest a score could be was -4. Hence, by default, most participants were reluctant to share information. This reflects prior work on sharing fake news <cit.>.  Unlike other intervention studies, we did not filter out participants who did not share regularly on social media <cit.>.





Prior trust in media organizations moderated the effectiveness of warnings. As shown in model 1 of Table <ref>, participant's prior perceptions of the ability, benevolence, and integrity of news organizations and social media platforms significantly impacted trust in false information. Namely, those who had higher trust in news organizations had lower trust in false information (p < 0.001), and those who had higher trust in social media had higher trust in false information (p < 0.001). Sharing intentions were related to trust in social media platforms but not news organizations. Participants who trusted social media more, reported higher sharing intention of false information (p < 0.05). 

By examining the interactions between prior media trust and each condition, we can explore these effects further.  In Table <ref>, we show the results of three Ordinary Least Squares (OLS) regressions on participant-level trust. The goal of these simpler models was to isolate the interaction effects from participants' prior media trust and political leaning on each condition (We discuss political leaning in the next section). In Figure <ref>, we show the interactions of participant-level trust across prior media trust subsets, where the high ABI group (for news or social media) contained participants who scored zero or above according to our compressed ABI scores. The low ABI group contained participants who scored below zero. 

Within these participant subsets, warners differ in effectiveness. For instance, participants who trusted news organizations less, trusted warning labels from social media platforms less  (Figure <ref>a). In Table <ref>a, the interaction between prior trust in news organizations and warning labels from the platform were significant (p < 0.01). Furthermore, according to both a multilevel model and Tukey's HSD Test on this subset, warning labels from the platform did not significantly shift trust in false information for those who did not trust news organizations.  This subset of participants also trusted warnings from fact checkers and the crowd less (see Figure <ref>c, Figure <ref>g, and Table <ref>a). While there still was a significant shift towards distrusting false information when labels from fact checkers or from the crowd were shown compared to control, the effect size between control and each condition was less than half of the effect size between control and the AI condition (0.213 for Fact Checkers and 0.198 for Crowd versus 0.471 for AI). In fact, for participants who trusted news organizations less, warning labels from AI were significantly more effective than all other warning labels (p < 0.01 for all pairs according to a multilevel model on this subset). On the other hand, for participants who trusted news organizations more, this pattern was not found. Warnings from AI were no more effective than warning labels from any other source and all warning labels significantly shifted trust relative to control.

 Perhaps expected, prior trust in social media platforms significantly moderated the effectiveness of warnings from the platform (p < 0.001 in Table <ref>). In fact, warning labels from the platform did not significantly shift trust relative to control for participants who trusted social media platforms less (p = 0.078 according to Tukey's HSD). Warnings from the crowd and fact checkers were also significantly moderated by prior trust in social media platforms. Saliently, participants who trusted social media platforms less, trusted warning labels from AI modestly more than other warning labels (p < 0.05 at post-level Tukey's HSD), while labels from AI were no more effective than other labels for those who trusted social media platforms more.

 A similar pattern was true across both prior media trust metrics for sharing intentions. Prior trust in news organizations significantly interacted with warning labels from the platform, with those who trusted news organizations less being less impacted by those labels. Prior trust in social media platforms significantly moderated the effect of all warning labels on sharing intentions, with those who trusted social media platforms less being less impacted by all labels. Due to space restrictions, we show the interaction analysis for sharing intentions in the supplemental materials.

 Political identity did not robustly moderate the effectiveness of warning labels. Despite the information presented not always being political in topic, self-reported political leaning still influenced trust in false information. As shown in Table <ref>, the more liberal a participant claimed to be, the less they trusted in false information (p < 0.001). This result may be in part due to the imbalance between conservative and liberals in the study population; however, in general, this finding aligns with the results of prior studies <cit.>.

In Figure <ref>, we show the distributions of participant-level trust in false information across three groups of participants: conservatives, moderates, and liberals. From these sub-groups, we found some evidence that conservative participants were less affected by warning labels from the platform. Namely, the effect size between control and Platform condition for conservative participants was less than half of the effect size for moderates and liberals (0.187 for conservatives, 0.457 for moderates, and 0.462 for liberals). Further, according to Tukey's HSD, conservative participants were impacted by warnings from AI significantly more than warnings from the platform  (p = 0.038 at the post-level after Sidak-Bonferroni correction), while warnings from AI were not significantly different than any other warning source for liberal participants.  However, in our interaction analysis at the participant-level, political leaning did not significantly play a role in the effectiveness of differing warning labels (see Table <ref>c). Across all political subgroups, all warning sources were still significantly better than control. These patterns held true when asking about sharing intentions (Interaction analysis can be found in the supplemental material).












Information processing still played an important role.
While there was a clear impact of warning labels on trust in false information, the information itself was still important. Based on the large body of work on information processing and more recent work on warning labels, this role should be expected <cit.>. As indicated in Table <ref> model 1, participants who scored higher on the cognitive reflection test (measuring tendency to engage in further reflection) and were more educated, trusted false information less (p < 0.001 and p < 0.007, respectively). These two factors suggest that the information itself continues to be important in trust decisions through deep information processing, information familiarity, and topical expertise. Although, notably these same factors (cognitive reflection and education) did not significant impact sharing intentions.



To further show the role of information in trust, we computed a Chi-squared Test of Independence between each post and the frequency of trust on the 5-point Likert scale. Across all conditions, we found a significant relationship between posts and the frequency of trust (all with p < 0.001). When examining this frequency of trust across posts, each post had a different baseline trust across the population, and that trust shifted towards distrust when warning labels were attached. Even though the changing skew across conditions was clear, individual posts that were more trusted in the control condition continued to be more trusted in the warning label conditions. However, the relationship between posts and sharing intentions did not have significant a relationship, further suggesting that information processing occurred more in trust decisions than in sharing decisions. Details about the headlines used can be found in the supplemental material.
















§ FACTUAL MANIPULATION TEST

After the main trust task, we asked participants if they saw any warning labels during the survey and if so, which headlines. This manipulation test is done to ensure our intervention was correctly received by the participants. Overall, approximately 90% of participants in each warning label condition said they saw warning labels (91% in No Label, 89% in Platform, 92% in AI, and 91% in Fact Checkers). When digging deeper into the headline/label recall, on average participants across all conditions recalled 60% of the headlines with attached warning labels correctly (σ 0.425, x̃ 0.75). These results are slightly better than but still reflect the factual manipulation results found in <cit.>, where many users did not recognize that they were presented with warning labels. In our case, the vast majority of participants recognized warning labels, but recalling what headlines were paired with warning labels soon after the task was done less well. 






§ DISCUSSION

In this study, we examined the impact of warning label sources on the trust in and sharing intention of false information. For the most part, the results of this work support the notion that warning labels are generally effective at changing information behaviors <cit.>. In our study, we found that warnings from AI had the largest effect on trust in and sharing intentions of false information, followed by warnings from fact checkers. This efficacy ranking differs from prior work, which found that warnings from fact checkers were most effective and warnings from AI were least effective <cit.>. Notably, this prior work focused only on sharing intentions, and the impacts on sharing intentions across warners in our study were very small. These small differences mean that who the warner was didn't seem to change sharing intentions much.  However, warning source did weakly impact trust in our study. Specifically, warnings from AI decreased trust in false information compared not only to the control condition, but also compared to warnings from the crowd and the platform. Still, the differences in effectiveness across warners, no matter the metric, were small.

This efficacy difference between warnings from AI and warnings from other sources did not hold across all participants. Warnings from AI were no more effective than any other warning label for participants who reported a high trust in news organizations. On the other hand, warning labels from AI were significantly more effective than all other warning labels for participants who reported a low trust in news organizations. We also found weak evidence that self-reported political leaning impacted the effectiveness of warners. In particular, we found that the effect size between the control condition and the warning labels from the platform were more than double in size for liberals and moderates compared to conservatives.  However, political leaning did not significantly interact with the effectiveness of any warner. Perhaps more importantly, all warners significantly decreased trust in false information compared to control, no matter a participants political leaning.



For those who distrust news organizations, AI may be seen as a neutral party, particularly compared to sources that involve other humans (i.e., crowd, fact-checkers, platform). This explanation is supported by the open-ended reasoning from our experiment. Multiple participants explicitly stated trusting in AI more than other sources. For example, one participant said: “I honestly trust an AI more than mainstream media”. Others stated that they trust in AI as a news veracity tool. For example, one participant said: “I trust that AI is a good at confirming the validity of a story” and another stated: “I think AI would be great at determining if its fake or not”. When ranking the most frequent bigrams used in the open-ended reasoning across conditions, we found that “the AI” was the most mentioned bigram in the AI condition (mentioned 418 times), while in the other conditions, the most mentioned bigrams were more generic and did not mention the source of the warning, such as ”the warning” (mentioned 453 times across all warning label conditions). These results directly reflect the broader findings of prior work on the perceptions of AI in news, which demonstrated that individuals with lower media trust and politically right-leaning individuals had more positive views on AI's use in news <cit.>. 


Implications: The results of this work have implications for both methods and practice. First, choosing a metric of intervention effectiveness is important when designing experimental studies. We found that trust in false information and sharing intentions of false information were only weakly correlated with each other and that baseline sharing intentions were very different than baseline trust in information. To fully capture the effects of warning label interventions, or any content moderation intervention, studies should use multiple target metrics.

Second, we found that the information itself played a significant role in trust decisions, although not in sharing intentions. This result again suggests that the intervention metric matters, but it also suggest that information selection matters. As with other warning label studies, the hope is that with a large enough headline sample, the impact of individual pieces of information on the larger pictures results will be negligible. However, future work should explore the relationship between information and interventions. From prior research, we know that political congruence with information may matter in intervention effectiveness. For instance, <cit.> found that if rumors are debunked by an unlikely partisan source, they were more effective (i.e., a CEO of a fossil fuel company correcting misinformation about climate change <cit.>). We also know many things about information processing and trust, as described in the Related Work section of this paper. However, continued exploration of how these factors interact with warning labels is critical to fully understand best design practices.

Third, this work adds to the already growing evidence that AI-produced content labels can be effective <cit.>. The implications of this result are mixed. Given that warnings from AI were significantly effective across media trust groups and political groups in our study, AI may be a perceived as a trusted neutral source for veracity information. However, this result should also point to the continue need to ensure when warning labels are used, they are used correctly. This caution is especially strong when using automated methods to determine when warning labels should be attached, as they are prone to error <cit.>. If warnings from AI are broadly effective but incorrect, information consumers may be harmed. Future work should continue to establish the consequences of using automated content moderation, both when those methods are correct and when they are wrong <cit.>. 



§ LIMITATIONS AND FUTURE WORK

First, it is important to recognize that these results may eventually change. Memory and attention, both collectively and individually, change over time <cit.>. For example, trust in Twitter has changed as ownership has changed <cit.>. Likewise, one can reason that as AI applications improve and people experience AI in more areas of their lives, they may gain trust in other AI tools. Or conversely, as people experience AI making mistakes in more areas of their lives, they may lose trust in other AI tools. Similarly, while there is currently high distrust in fact checkers by the U.S. political right <cit.>, as the environment around fact checking and politics changes, so too may trust in them by particular groups (although as suggested by both this study and others, warnings from fact checkers still benefit those who distrust them <cit.>). Future work should continue to examine the relationships between warning efficacy and consumer's prior experiences.  Further, future work should focus on longitudinal studies given the evolving nature information environments.

 Second, it may be possible that our metric questions influenced each other. That is, we asked each participant about their trust in the information before asking about if they would share the information. Given that accuracy prompts influence sharing rates <cit.>, it may be possible that asking consumers to think about their trust in information decreases their intention to share that information. Future work can test this potential limitation through separated, but comparable, studies of the two concepts.

 Third, a key limitation with any controlled experiment is the inability to perfectly simulate passive information consumption as one may do when scrolling through a news feed. While our study ensured participants were not primed to actively consume warning labels, this limitation still exists. Further, given the simulated setting, there are no social consequences for sharing or not sharing information <cit.>. These active and passive consumption differences may partially change the effectiveness of warning labels and the efficacy differences from varying warners. These results should be thought of as the impact of warners when consumption is at least somewhat active.  While we may be able to do slightly better at simulating passive browsing by building dynamic sandbox environments, these limitations will, at least to some extent, still exist, as participants know they are in a study environment. Better understanding the relative knowledge gains from constructing more realistic testing environments is an avenue for future research.

 This experiment, along with the majority of experiments currently done in this space, should be considered somewhere between `Stage 1' and `Stage 2' of intervention development (as described in the NIH Stage Model for Behavioral Intervention Development <cit.>). This means we are between refining existing intervention designs and testing the efficacy of those designs in research settings (where our key limitations exist). The next stage (`Stage 3') for intervention work is efficacy testing with real-world platforms. Unfortunately, this stage of research is often left to the platforms themselves, and occasionally select academics. Hence, it may not always be possible for research to get to this stage. Although, we would argue that having some barrier to entry to run experiments on real platforms is important to ensure harms are not caused by poorly designed and poorly debriefed experiments. Fortunately, recent intervention research that has made it to Stage 3 suggests that the results from lab experiments (like the one in this paper) may hold in real-life systems. In a working paper by <cit.>, accuracy prompt advertisements were deployed to 33M Facebook users and 75K Twitter users. The results from this at-scale experiment showed that accuracy messages reduce
misinformation sharing, aligning with the results previously found in research settings.





§ CONCLUSION

In this work, we sought to better understand the impact of content warning label sources on the effectiveness of those labels, both in terms of trust in and sharing intentions of false information. Using a five-condition between-subjects experiment, we found that, compared to having no warning labels, all warnings significantly shifted trust in false information but not all warnings significantly shifted sharing intentions of false information. Still, with few exceptions, this work supports the view that content warning label interventions are generally effective. Most notably, we found that warnings from AI were slightly more effective overall than other warners.  That is, warnings from AI were effective across both metrics (trust and sharing), across prior trust attitudes, and across participant political leaning. While prior trust in news organizations moderated the effectiveness of warnings from the platform, the crowd, and fact checkers, it did not significantly moderate warnings from AI. Overall, we think this key result is both promising and concerning. While on one hand, our results show the potential usefulness and neutrality of AI in news veracity tasks, it also implies the need for highly accurate AI tools for news veracity tasks, given their power in changing information behaviors. Together, we hope the results of this work inform both the design of content labels on real-life systems and future studies of veracity-based information interventions.








§ PAPER CHECKLIST





  * For most authors...

    
  * Would answering this research question advance science without violating social contracts, such as violating privacy norms, perpetuating unfair profiling, exacerbating the socio-economic divide, or implying disrespect to societies or cultures?
    Yes. The research questions answered in this paper advance research in content moderation and labeling, without violating social contracts. All data collected through Prolific was done so anonymously with IRB approval and with fair pay. Further, given the use of real, fact-checked information, we ensure that all participants are debriefed about the information's veracity.
  
  * Do your main claims in the abstract and introduction accurately reflect the paper's contributions and scope?
    Yes, the claims made throughout the paper have been carefully reviewed and contextualized. We ensure that the claims made in the abstract and the introduction accurately reflect the results of the paper.
   
  * Do you clarify how the proposed methodological approach is appropriate for the claims made? 
    Yes, we have worked hard to ensure each methodological step is justified and appropriate for the claims made. The details of our methodological approach can be found in the Section entitled “Methods”.
   
  * Do you clarify what are possible artifacts in the data used, given population-specific distributions?
    Yes, we clarify where the imbalance in the population sample may impact the results of our analysis. We discuss the population sample in the subsection entitled “Data Sample and Demographics”. In addition, we provide details about all demographic distributions captured in the study in our supplemental materials document.
  
  * Did you describe the limitations of your work?
    Yes. We ensure that our work is contextualized by its limitations. Our discussion of the key limitations of the work can be found in Section “Limitations and Future Work”.
  
  * Did you discuss any potential negative societal impacts of your work?
    Yes. In particular, we discuss how our results have both positive and negative potential implications for the use of warning labels in real-life systems. More generally, we do not create any artifacts that can be used outside of this work.
      
  * Did you discuss any potential misuse of your work?
    No. We do not curate or release any resources, data, or software that have potential for misuse.
    
  * Did you describe steps taken to prevent or mitigate potential negative outcomes of the research, such as data and model documentation, data anonymization, responsible release, access control, and the reproducibility of findings?
    Yes. We use best practices for both human experiments and for misinformation research, including anonymization and debriefing. As described in the Section “Methods”, we describe each step taken in this work to ensure it is reproducible. Further, we additional information and data in our supplemental materials to ensure that the work is reproducible.
  
  * Have you read the ethics review guidelines and ensured that your paper conforms to them?
    Yes. We have carefully reviewed the ethics review guidelines and ensured that this paper conforms to them.



  * Additionally, if your study involves hypotheses testing...

  
  * Did you clearly state the assumptions underlying all theoretical results?
    NA
  
  * Have you provided justifications for all theoretical results?
    NA
  
  * Did you discuss competing hypotheses or theories that might challenge or complement your theoretical results?
    NA
  
  * Have you considered alternative mechanisms or explanations that might account for the same outcomes observed in your study?
    NA
  
  * Did you address potential biases or limitations in your theoretical framework?
    NA
  
  * Have you related your theoretical results to the existing literature in social science?
    NA
  
  * Did you discuss the implications of your theoretical results for policy, practice, or further research in the social science domain?
    NA



  * Additionally, if you are including theoretical proofs...

  
  * Did you state the full set of assumptions of all theoretical results?
    NA
	
  * Did you include complete proofs of all theoretical results?
    NA



  * Additionally, if you ran machine learning experiments...

  
  * Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    While we did not run machine learning experiments, we do document the statistical model implementation used to analyze the data in the capture of Table <ref>. Further we provide details of our model choice in the Section “Methods”.
  
  * Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    NA
     
  * Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    NA
	
  * Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
    NA
     
  * Do you justify how the proposed evaluation is sufficient and appropriate to the claims made? 
    Yes. We carefully thought through and justified the choice of statistical model based on the structure of the data collected. This justification is described in the Section entitled “Methods”.
     
  * Do you discuss what is “the cost“ of misclassification and fault (in)tolerance?
    NA
  



  * Additionally, if you are using existing assets (e.g., code, data, models) or curating/releasing new assets, without compromising anonymity...

  
  * If your work uses existing assets, did you cite the creators?
    NA
  
  * Did you mention the license of the assets?
    NA
  
  * Did you include any new assets in the supplemental material or as a URL?
    NA
  
  * Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    NA
  
  * Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    NA

  * If you are curating or releasing new datasets, did you discuss how you intend to make your datasets FAIR?
NA

  * If you are curating or releasing new datasets, did you create a Datasheet for the Dataset? 
NA



  * Additionally, if you used crowdsourcing or conducted research with human subjects, without compromising anonymity...

  
  * Did you include the full text of instructions given to participants and screenshots?
    Yes. We have provided details about how the experiment was conducted (in Section “Methods”) and we provided screenshots of what participants saw during the study. In addition, we provide the full survey in the supplemental materials.
  
  * Did you describe any potential participant risks, with mentions of Institutional Review Board (IRB) approvals?
    Yes. The experiment was IRB approved and participants were provided with a consent form that outlines risks and benefits of participating in the study.
  
  * Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    Yes. The hourly wage paid to participants and the estimated time of completion was provided to participants prior to the study. Further, participants were free to stop the study at any point and were paid for completing at least 50% of the survey. This payment amount is described in the paper.
   
  * Did you discuss how data is stored, shared, and deidentified?
   Yes. All data was collected and stored anonymously. Participants were told this in advanced. 








 Paper Title: The Impacts of AI Avatar Appearance and Disclosure on User Motivation.







AI and Avatars






B. Visser et al.



Leiden Institute of Advanced Computer Science, Leiden University
 P.O. Box 9512, 2300 RA Leiden, The Netherlands



The Impacts of AI Avatar Appearance

and Disclosure on User Motivation
    Boele Visser10009-0002-7504-6495 Peter van der Putten10000-0002-6507-6896 Amirhossein Zohrehvand10000-0002-7473-6702
    August 5, 2024
========================================================================================================================



§ ABSTRACT 
  This study examines the influence of perceived AI features on user motivation in virtual interactions. AI avatars, being disclosed as being an AI, or embodying specific genders, could be used in user-AI interactions. Leveraging insights from AI and avatar research, we explore how AI disclosure and gender affect user motivation. We conducted a game-based experiment involving over 72,500 participants who solved search problems alone or with an AI companion. Different groups experienced varying AI appearances and disclosures. We measured play intensity. Results revealed that the presence of another avatar led to less intense play compared to solo play. Disclosure of the avatar as AI heightened effort intensity compared to non-disclosed AI companions. Additionally, a masculine AI appearance reduced effort intensity.






§ INTRODUCTION


With the rise of digital worlds <cit.>, humans are increasingly interacting with different forms of Artificial Intelligence (AI) in digital spaces, for example, in the form of the recommended system in social media, chatbots or avatars. Similar to the offline world, the perceived gender of the entity that we are interacting with shapes our behaviour and motivation <cit.>. Past research on motivation in offline settings argues that gender differences in motivation arise from the social and cultural expectations, experiences and roles that are associated with being male or female <cit.>. Gender roles and stereotypes can shape an individual’s beliefs, values and behaviours, including their motivation <cit.>. These predispositional features are almost immutable.

However, contrary to the offline world, the gender of the agent that we are interacting with, i.e. the AI agent, is customizable. The AI gender appearances could be chosen on a scale, allowing us to experiment with immutable characteristics and observe their impact on user behaviour. Building on social exchange theory <cit.> and trust and agency literature <cit.> we argue that users playing together with an avatar exhibit a higher degree of persistence in comparison with playing alone. Further, we argue that users are more motivated to play with an agent if they attribute more agency to or perceive the gender of the agent to be of a more cooperative type, they exhibit higher degrees of persistence. Therefore, we suggest that users play less intensely when they are playing with an AI in comparison to a human. Likewise, we argue that the disclosure that an avatar is actually AI-driven reduces the intensity which with the user plays, just as what happens when AI systems in the physical world are disclosed  <cit.>.

To research this, we performed an experiment in a game. In the game, published on Roblox  (a virtual world), players have to perform a simple task, which can be done together with others. Inside this game, over 72,500 participants played 15 levels of a simple search task. The participants either played alone or together with an AI avatar, from which the gender appearance was randomly chosen. For half of the participants playing together with an AI, it was not disclosed that they were playing with an AI. We measured how intense participants played the experiment by dividing the time before leaving the experiment by the number of levels completed. We used this measurement to compare the difference in motivation between different treatment groups, to see what effect the AI gender appearance has on motivation in this particular context. The findings of this study expand our understanding of social exchange theory in the online world <cit.> and trust in AI <cit.>  by showing how the perceived gender of a non-human agent influences human motivation and trust.



§ BACKGROUND

Motivation is defined as “the degree and type of effort that an individual exhibits in a behavioural situation” <cit.>. There are two types of effort: intrinsic motivation and extrinsic motivation <cit.>. Intrinsic motivation occurs when an individual finds something interesting or enjoyable, while extrinsic motivation occurs when an individual is motivated by the outcome of a task. While intrinsic motivation is more about the process, extrinsic motivation is more about the outcome. Past research shows that different demographics show categorically different motivational patterns or the demographics of others might influence their motivation. According to a literature review <cit.>, motivation drivers vary among age groups.



 §.§ Social interactions with AI

Interactions with artificial intelligence (AI) are becoming increasingly common.  Considering that users can give agency to the AI agent, the characteristics of the AI can influence how people interact <cit.> and trust AI based on their ability to think, plan and act <cit.>. Past research shows that even the features of how the AI agent is described to users can influence the interaction and the user <cit.>. Within the interaction itself, the form factor (i.e. written, voice, physical) of the AI changes how participants conform to the AI <cit.>. In these interactions, the perceived gender of the AI influences the compliance with the AI <cit.>. To sum, past research shows that various factors influence how people interact with an AI. Increasingly, virtual worlds are where many of these interactions with AI could take place.

Virtual spaces are expanding beyond gaming communities to offer courses, stores, and financial services, allowing consumers to discuss their interactions with firms, and for many more aspects of everyday life to take place online <cit.>.  Both intrinsic and extrinsic motivation, measured by factors such as escapism, visual attractiveness, perceived usefulness, and entertainment value, impact users' attitudes towards using virtual worlds <cit.>.

In virtual worlds, users are typically represented by avatars, which are graphical representations of themselves <cit.>. Users can design their avatars to represent their actual appearance or their desired identity in the virtual world <cit.>.
The appearance of an avatar can influence the behaviour and motivation of the user, especially when the user can identify with their avatar <cit.>. The appearance of an avatar can also influence how a user is perceived by others in virtual worlds <cit.>. Particularly, avatar gender can also influence how a user is perceived and behaves in virtual worlds <cit.>.

Users, similar to the offline world, can interact with others, represented by some avatar, in a virtual world. Past research suggests that cooperating with other players motivates players to use a virtual world: When an educational video game app added a feature where users could asynchronously cooperate, the app usage increased 217% <cit.>.
Synchronous cooperation also motivates players: players of video games are motivated to play an exergame again more when they play together (in cooperation) with another player in the same physical space than when they played alone <cit.>. When the AI behaves like a player, or when the player cannot distinguish an AI agent from a human player, there should be no difference between the interaction with a player and with an AI, thus the effects of cooperating with another human could carry over to cooperating with an AI. Therefore, we suggest the following:

H1: Participants who play with an AI avatar (treatment group, regardless of disclosure) have a higher intensity of playing than those who play alone (control group).



 §.§ Relationship between AI avatar gender and motivation

The gender of others who someone interacts with influences motivation <cit.>. Some research claims that men are motivated by competition and they believe they outperform others <cit.>, so men will want to keep ahead of others and are motivated to do so quickly. Other research claims that women behave more prosocially <cit.>, so women might want to support others and continue to keep interacting longer, which will make them play longer.

Since an AI is perceived to be a social actor by humans <cit.>, the characteristics of the AI determine how humans interact with the AI <cit.> and the perceived gender of an AI having an influence on people’s motivation during textual interactions <cit.>, the gender effects may carry on when interacting with an AI. We therefore suggest the following:

H2: Participants playing with an AI avatar that appears more masculine have a higher intensity of playing.



 §.§ Relationship between AI disclosure and motivation

When interacting with an AI, the user could know whether the one they are interacting with is an AI or not (which implies it is a human). This AI disclosure affects how users perceive an AI system <cit.>. Employees who get performance feedback created by an AI get a negative perception of this feedback when they know the feedback is created by an AI, which makes them less motivated by the feedback <cit.>. Additionally, Job candidates are less interested in an offer when their application is reviewed by an AI system and therefore are less motivated <cit.>. Note that the framing that is used when introducing the AI may have an influence on trust and conformance <cit.>). This lets us believe that AI disclosure causes a lower motivation of users, and therefore we suggest the following:

H3: Intensity of playing is negatively associated with participants knowing that they are playing with an AI avatar, in comparison to playing without the AI being disclosed.



§ METHODS

In this section, we will discuss the methodology of the experiment we performed. We preregistered this experiment at the Open Science Foundation.



 §.§ Organizational setting

The setting for this experiment is a game on Roblox. Roblox is an online gaming platform that allows users to create, share and play their own virtual experiences. This platform has over 150 million monthly active users. The game had more than 50.000 active daily users at the time of the experiment. The idea behind the game is simple: players get placed in a level (a room), and their task is to find a hidden object inside that room. Once they find the object they click on it, and they progress to the next level. 

The player base of the game is very diverse. Players come from around the world, with different age groups, languages and devices. This makes it a good fit to research the impact of an avatar’s gender on motivation in a very diverse player base.



 §.§ Participants

Participants are players of the Game X. Players could join the experiment by selecting a special mode called ‘Science Mode’. Only the data from plays during the first two weeks are used, although it is possible to join the ‘Science Mode’ after the experiment’s duration to be able to review the procedure. After participating in the experiment, players receive a cosmetic item and some in-game virtual currency.

Participants could participate in the experiment multiple times, but only their first participation will be taken into account for the analysis.

Due to the privacy regulations of Roblox, it is not possible to review the age and gender of the participant. However, data about age is available on a generic level of the game, where we could see the distribution of players in age groups. It is most likely that the participants follow the same demographics as the game itself. Players from the game come from around the world, which makes the participants very diverse and eliminates the WEIRD society bias from the data <cit.>.



 §.§ Procedure

When participants join the experiment, they get an information screen that they are going to participate in an experiment. One-third of the participants are in the control group and will participate alone. The other two-thirds are in the subject group and will participate in an AI avatar. Within the subject group, one-half was told they were playing with an AI (so one-third of all participants), while the other half (also one-third) was not told they were playing with an AI.

After the explanation, participants select their avatar. Participants were completely free to select the avatar they wanted; the question asked was ‘Select your avatar’. The avatars differ in gender appearance. The avatars were always displayed in the same order, from left to right (See Figure <ref>). Participants play with their selected avatar during the experiment, and participants are asked to rate this avatar on a 5-point scale ranging from ‘very feminine’ to ‘very masculine’.




         







        






After they select the avatar, they play the ‘Science Mode’, which consists of 15 stages. In this mode, players cannot see other participants, except the AI avatar controlled by the computer when participants are in the subject group. This AI avatar is also called a non-player character (NPC).

The appearance of the NPC is one of the 5 different avatars shown during the self-selection process. The appearance of the NPC is selected randomly out of the five preset avatars, regardless of the avatar chosen by the player. The NPC wanders around the room to preset positions and pretends to be searching for the button. When the participant finds the button, the NPC teleports to the next level within a few seconds. Therefore, the NPC is always playing at the same level as the participant, which gives the participant the feeling that the NPC is playing with them.

The difficulty and level design is the same for all participants, as is the behaviour of the NPC. The independent variables that are changed are playing alone vs. playing with an NPC, the disclosure of the NPC being an AI and the (gender) appearance of the NPC.

Before joining the experiment, participants are informed about its details. The experiment is non-intrusive and participation is voluntary, with the option to leave at any time without giving a reason. Participants are informed about the treatment type directly after participation. The AI is programmed to never harm participants, but rather to encourage and help them to complete the task while remaining friendly at all times. No personal identifier data was stored about the user (data collection has been fully anonymized).



 §.§ Measures

We proxy motivation based on how intensely players engage with the experiment and how likely it is that they complete all the stages. We measure the seconds spent per level by calculating the time before leaving the experiment divided by the number of levels completed. A lower value of seconds spent per level therefore implies that the participant spends less time on a level on average, and therefore engages more intensely with the experiment. The motivation of the participants is measured by using a performance-based behavioural measure. Performance-based behavioural measures indicate both a higher outcome-focused motivation (extrinsic motivation) and process-focused motivation (intrinsic motivation) <cit.>.

In addition, we use a dummy variable to indicate if a participant has completed all 15 stages. The time before ending the participation is also measured - participation can end by either completing the 15 stages or leaving earlier. The speed at which players complete the experiment therefore indicates if the avatar's appearance influences either the intrinsic process-focused motivation or extrinsic outcome-focused motivation.

To get a more reliable measure for the gender of the avatars, we asked each participant to rate their avatar on a 5-point scale, with 1 being ‘very feminine’  and 5 being ‘very masculine’. This gives an average score for each avatar between 1 and 5. To simplify the analysis, we calculated the average rating score of the avatar. Given that the average of our 1-5 scale is 3, we mapped the avatar to either feminine when the average score is below 3 or masculine when the average score is above 3. If a score is exactly 3, the avatar is treated as having no gender and is excluded from the analysis about gender.



 §.§ Analytical strategy

To test the hypotheses, we used Ordinary Least Squares (OLS) from the statsmodels package <cit.> in Python. For each variable, the dependent variable is the seconds spent per level, modelled using OLS. The independent variable is AI avatar presence, AI avatar masculinity and AI disclosure for respectively H1, H2 and H3.




§ RESULTS

During the 2-week time window, 126,013 participants started the experiment. From this group, 8488 participants (6.74%) completed all levels of the experiment.

In Figure <ref> the progress of participants over the experiment is displayed. In Figure <ref>, the time spent in the experiment before either leaving or completing the experiment is displayed. The mean time spent in the experiment is 249.4 (σ = 287.6), while the median time spent is 142 seconds. The mean time including the standard deviation is plotted in the error bar displayed in Figure <ref>.







53,099 (42.14%) participants left the experiment during the first level. This implies that participants did not finish a level. Since they did not finish a level, participants never completed a task. This data is left out of our analysis because there is no indication that participants did understand the task well enough or that their motivation was affected by the treatment they were subjected to.

Based on previous observations within the game, it is not possible to complete a stage within 5 seconds when you did not play the level before (based on loading times, time to search the button and the time to click on it). If a participant completed a stage within this 5-second timeframe, it is likely that the participant used some kind of cheating or did know the location of the button beforehand. Therefore, we left out the data of players from which the seconds spent per level (the average time spent per level) is below 5.0 seconds. This was the case for 11 (0.01%) participants. This makes the total sample for analysis consist of 72,903 participants, of which 8488 (11.64%) completed the experiment.

In Table <ref>, an analysis of the difference between participants who did and did not complete the experiment could be found. 11.6% of the participants completed all levels of the experiment, with the average participant completing 3.67 levels (when excluding participants who did not complete a single level).




As mentioned in the methodology, there is no information available about the age of individual participants. However, if we assume the participants in the experiment follow the same demographics as the game, nearly 50 per cent of the participants are 13 or younger, with an additional 17 per cent being aged between 13 and 17. There are no other game-level demographics available.

Out of the participants, 23,079 (31.66%) participants were in the control group, 24,894 (34.15%) participants played together with an AI and did not know they were playing with an AI and 24.930 (34.20%) participants played together with an AI and knew they playing with an AI. The distribution of both the self-chosen avatar and the assigned AI avatar of the groups who played with an AI are listed in Table <ref>. Over 50% of the participants chose the first avatar as their avatar.



To assess the gender of the avatars, we took the mean of all rankings given for that particular avatar. If the average avatar score was under 3, it was categorized feminine, while it was categorized masculine when the average avatar score was over 3. The categorization of all avatar types is listed in Table <ref>.





 §.§ Hypothesis Tests

We conducted an ordinary least squares regression analysis to estimate the relationship between the presence of an AI avatar (treatment group) and the seconds spent per level (H1: Participants who play with an AI avatar (treatment group) have a higher intensity of playing than those who play alone (control group)). In Figure <ref>, the seconds spent per level grouped by the treatment type is displayed.



We found a significant effect (R2 = 0.01, F(1, 72901) = 734.4, p < 0.001, 95% CI [12.033, 13.909]) for intensity of play. Participants’ seconds spent per level increased by 13.0 seconds per level when playing together. That is equivalent to 0.22 standard deviations. This means that participants who played together with another (AI) avatar played less intensely. This is the inverse of what we expected, and therefore we reject H2b.

To test the influence of the effect of the gender appearance of the AI avatar, we conducted an ordinary least squares regression analysis (H2: Participants playing with an AI avatar that appears more masculine have a higher intensity of playing).

We found a significant effect for seconds spent per level (R2 = 0.000, F(1, 72901) = 10.03, p = 0.002, 95% CI [0.841, 3.570]). Participants’ seconds spent per level increased by 2.2 seconds per level when playing together with a more masculine AI avatar. That is equivalent to 0.04 standard deviations. This means that participants who played together with an AI avatar that appeared more masculine played less intensely. This is the inverse of what we expected, and therefore we reject H2.

To test the hypothesis about the effect of participants knowing that they are playing with an AI avatar on their intensity and duration of play, we conducted an ordinary least squares regression analysis (H3: Intensity of play is negatively associated with participants knowing that they are playing with an AI avatar). In Figure <ref>, the seconds spent per level grouped by the treatment type is displayed.



We found a significant effect (R2 = 0.000, F(1, 49822) = 6.957, p = 0.008, 95% CI [-2.569, -0.379]) for seconds spent per level. Participants’ seconds spent per level decreased by 1.5 seconds per level when the other avatar was disclosed as AI. That is equivalent to 0.02 standard deviations. This means that participants who played together with an AI avatar that was disclosed as such, play more intensely. This is opposing to what we expected.

In addition, we did not find a significant effect (R2 = 0.000, F(1, 48007) = 0.803, p = 0.370, 95% CI [-0.003, 0.008]) that playing together with an AI avatar disclosed as such increases the value of seconds spent per level, compared to playing alone.




§ DISCUSSION

This study researched various independent variables that could be used by an avatar controlled by Artificial Intelligence (AI) to improve users’ motivation. Our findings show that in our experiment and game, the masculinity of the users’ avatar makes participants play more intensely. Furthermore, playing together with another avatar controlled by AI significantly affects users’ motivation. Besides that, the masculinity of the AI’s avatar significantly increases the intensity of play. When users learn that the other avatar is controlled by AI, users had a higher probability of not completing the experiment and played more intensely. Finally, the difference between the user’s gender appearance and AI’s avatar gender appearance moderates the effect of the AI’s avatar gender appearance and makes users play more intensely.



 §.§ Contribution to existing literature

Previous studies have shown that cooperating with others can increase motivation <cit.>. Our research shows that playing together with an avatar controlled by an AI makes players play more intense. Our findings suggest that the existing theory, which states that playing with someone else increases motivation, can be extended to include playing with an avatar controlled by AI. Future research in this field could focus on replicating our findings in other games and bringing these effects to non-gaming virtual worlds, to see if our findings are also applicable to other settings, such as virtual workspaces.



 §.§ Practical implications

Companies that create virtual worlds, such as Meta, could use this research when designing avatars in the virtual world. If the sole goal would be to maximize player intensity, it would be best to use a masculine avatar for the AI and to tell users that they are interacting with an AI, in order to motivate users in their virtual world. However, the results of this study also show the need for diversity in virtual worlds. We have seen that the similarity between the participants’ avatar gender appearance and the AI avatar gender appearance moderates the effects of the AI avatar gender appearance on motivation. Therefore, preset genders of both the own avatar gender and the AI avatar gender can limit the effect of the AI avatar on motivation. Diversity in both avatars, such as being able to choose the avatars from a diverse list or being able to freely create the avatars, could help the user in motivation within the virtual world. Beyond gender, ensure that the avatars communicate attributes players are looking for, especially if you want to promote gender diversity in AI collaborators, whilst still stimulating motivation. This requires more research to investigate what other attributes might affect the motivation of players, as well as a broader coverage of the gender continuum, for instance by introducing more androgynous characters.



 §.§ Limitations

Our AI avatar movement was simple. The avatar moved to preset points and did not interact with participants in another way, such as talking or following them. This could have limited the degree to which participants bestow agency to the AI. Furthermore, we don't theorize or study how the details of the interaction could change the user's perception regarding the agency of the AI  <cit.>. To address this limitation, future research could use a more naturally behaving AI, introduce different levels of intelligence, and observe the impact on the user's behaviour.

Besides the quality of the AI movement, the task and its setting might have influenced the outcomes of the study. The task was a simple search task, which could be seen as a more mathematical task, and the levels were themed around science. This particular task and setting could have led to a higher base motivation of some groups of participants, which could have worked through the measurements. Future research could be changing the task and setting, while keeping the rest of the methodology the same, to investigate if the task and setting influences the effects we found.

Finally, the selection of avatars was limited to only five avatars, which restricted the gender expressions in the avatars and diversity in skin colour. A participant raised this concern, which did not feel represented by the available avatars due to skin color. We decided to not include diverse avatars because participants should choose based on gender, not on other avatar features. However, this non-diverse set of avatars could have led to dissatisfaction with the experiment and decreased motivation regardless of the treatment type for some participants. To address this limitation, future research could include a wider range of avatars to choose from or allow players to use their avatar.



§ CONCLUSION

This study aimed to describe how AI avatars inside virtual worlds could be used to increase the motivation of a user. We conducted a large-scale field study inside a game where we were able to control variables in the virtual world. We found that playing together with an AI avatar makes users play less intensely. When this AI avatar appears more masculine, users play less intensely. The AI avatar being disclosed as an AI make users play more intensely. Further research could include replicating these results in other games & non-gaming virtual worlds and researching what the causes are for the effects we found.





  §.§.§ 

Boele Visser is co-owner of the game in which the research took place. The other authors have no competing interests.









splncs04









 Paper Title: CREW: Facilitating Human-AI Teaming Research.



Gradient dynamics model for drops of volatile liquid on a porous substrate
    Uwe Thiele
    August 5, 2024
==========================================================================



§ ABSTRACT 
  With the increasing deployment of artificial intelligence (AI) technologies, the potential of humans working with AI agents has been growing at a great speed. Human-AI teaming is an important paradigm for studying various aspects when humans and AI agents work together. The unique aspect of Human-AI teaming research is the need to jointly study humans and AI agents, demanding multidisciplinary research efforts from machine learning to human-computer interaction, robotics, cognitive science, neuroscience, psychology, social science, and complex systems. However, existing platforms for Human-AI teaming research are limited, often supporting oversimplified scenarios and a single task, or specifically focusing on either human-teaming research or multi-agent AI algorithms. We introduce CREW, a platform to facilitate Human-AI teaming research and engage collaborations from multiple scientific disciplines, with a strong emphasis on human involvement. It includes pre-built tasks for cognitive studies and Human-AI teaming with expandable potentials from our modular design. Following conventional cognitive neuroscience research, CREW also supports multimodal human physiological signal recording for behavior analysis. Moreover, CREW benchmarks real-time human-guided reinforcement learning agents using state-of-the-art algorithms and well-tuned baselines. With CREW, we were able to conduct 50 human subject studies within a week to verify the effectiveness of our benchmark.







§ INTRODUCTION













Over the past decade, significant progress in machine learning has increased the potential and necessity for humans to collaborate and interact with Artificial Intelligence (AI) agents. Human-AI teaming has emerged as a research paradigm to explore the dynamic interactions and collaborative processes between humans and AI. By leveraging the complementary strength of both humans and AI, advancements can significantly enhance the overall team performance.

Unlike traditional AI research, which typically focuses on machine learning algorithms in isolation, Human-AI teaming requires a multidisciplinary approach to incorporate insights from various scientific domains. Numerous studies have examined human-human teaming <cit.> with cognitive science, neuroscience, and psychology. Machine learning and robotics communities have extensively researched multi-agent machine learning <cit.>, while team dynamics <cit.> has been explored in complex systems, social science, and network science. Despite the importance and potential of this research paradigm, there is still a lack of a comprehensive and unified platform to benefit research on joint efforts across disciplines and scalable hypothesis verification.

Developing a comprehensive platform for Human-AI teaming research presents several unique challenges. Firstly, the platform needs to support diverse tasks with varying complexities with easy extensions for new tasks or modifications. While reinforcement learning research platforms <cit.> have widely adopted this practice, current Human-AI teaming platforms remain limited to single tasks <cit.>. Secondly, enabling real-time communication through various modalities between multiple humans and heterogeneous AI policies is essential for effective collaboration. However, existing solutions typically support human feedback only through replaying offline trajectories and do not implement real-time feedback mechanisms. Understanding how to build AI that can team with, learn from, be guided by, align with, and augment humans is as crucial as modeling human behavior during interactions with AI. Therefore, the platform must provide interfaces for recording human physiological data alongside agent data, tailored for cognitive science and neuroscience studies. Furthermore, current studies often involve small participant groups, making it difficult to derive generalizable conclusions. Lastly, the absence of a unified platform has limited fully open-sourced studies and the establishment of appropriate benchmarks.

We present CREW, a platform designed to facilitate Human-AI teaming research aiming to address the above challenges. We develop CREW around key principles such as extensible and open environment design, real-time communication support, hybrid Human-AI teaming modes, parallel sessions for scalable experiments, and comprehensive human and agent data collection. Additionally, CREW incorporates highly modular algorithm components compatible with practices in the machine learning community. We demonstrate CREW's potential by benchmarking real-time human-guided reinforcement learning (RL) algorithms alongside various RL baselines. With CREW, we were able to conduct 50 human subject studies within a week. Moreover, CREW includes a set of cognitive tests to explore how individual differences among humans impact their effectiveness in training AI agents. To our knowledge, CREW is the first platform to unify the desired features for Human-AI teaming research across multiple disciplines. We aim for CREW to serve as an infrastructural foundation for multidisciplinary, reproducible, and scalable Human-AI teaming research.




§ RELATED WORK



Human-AI Teaming Research Extensive research has explored Human-AI teaming across various domains. Machine learning studies have developed algorithms to leverage human expertise to improve the accuracy <cit.>, robustness <cit.>, and interpretability <cit.> of models. Integrating human feedback can not only improve performance <cit.> but also align the models with human preference <cit.>. Human-computer interaction research has created interfaces <cit.> and workflows <cit.> that enhance collaboration between humans and AI, combining their strengths to achieve superior performance. Ethical research focuses on understanding and mitigating the societal <cit.>, ethical <cit.>, and technical <cit.> challenges of the rapid advancement and wide adoption of AI. Many fields, including neuroscience, healthcare, robotics, transportation, education, security, and accessibility, have shown growing interest <cit.> in Human-AI teaming. Overall, the broad spectrum of interests highlights the need for multidisciplinary collaboration to drive further advancements.

Human-AI Teaming Platform While significant progress has been made in Human-AI teaming research, there remains an absence of a comprehensive research platform. Overcooked Environment <cit.> is a simplified version of the original popular game to challenge human agents and AI agents in tasks that require close coordination and strategic teamwork. StarCraft II Learning Environment (SC2LE) <cit.> supports adversarial settings to allow Human-AI interaction and learning from human demonstrations. Rapid Integration and Development Environment(RIDE) <cit.> focuses on defense-related scenarios, emphasizing rapid development and integration of AI systems for operational purposes. In addition to real-time decision-making tasks, previous research has developed platforms <cit.> that focus on offline preference or rating settings where humans can provide offline evaluations or corrections with imitation learning or reinforcement learning.



However, existing platforms have more than one of the following limitations that constrain Human-AI teaming research as summarized in Tab. <ref>. Most environments only support one type of task that can be difficult to extend, and the interactions between humans and machine learning agents are limited to mouse and keyboard operations. Moreover, most of the environments only support the data collection on the game data, such as state, action, or reward focusing on the machine learning algorithm development, but none of them support the collection and analysis of human physiological data (eye movement, pupillometry, brain activity, heart impulse, or natural language) to understand human cognitive states and different effects on human subjects. Furthermore, the scale of collaboration has been limited to two agents in a collaborative or competitive setting. Notably, most existing Human-AI teaming studies have only been evaluated on a very small number of subjects or among the authors, which greatly limits our understanding of the state-of-the-art performance. There remains a large gap between the existing platforms and the desired environment to facilitate future research.

Real-Time Human-Guided Learning Most real-world decision-making processes require rapid decisions and adaptation in real time. Therefore, real-time human-guided learning is an essential topic in Human-AI teaming research. Previous research has focused on algorithm design to integrate real-time human feedback into policy training. TAMER and Deep TAMER <cit.> learn to predict human discrete feedback and utilize the feedback model as the value function for policy learning. COACH and DeepCOACH <cit.> addressed the inconsistency of human feedback by modeling it as the advantage function under an actor-critic framework. Recent progress has further refined these algorithms, addressing various challenges in human feedback integration, such as different feedback modalities  feedback stochasticity <cit.>, and continuous action spaces <cit.>. Other works have explored indirectly inferring human objectives from feedback or preference <cit.>. CREW provides an extensive environment for real-time human-guided learning with online training and feedback integration, human physiological data collection, and parallel distributed experiment support.





§ CREW: DESIGN AND COMPONENTS






 §.§ Platform Vision and Design Philosophy




We design CREW to facilitate Human-AI teaming research. Our vision is to construct a unified platform for researchers from diverse backgrounds, allowing them to join forces from human-AI interaction, machine learning algorithms, workflow design, as well as human analysis and training. To achieve this, CREW incorporates the following capabilities, as illustrated in Fig. <ref>.

Extensible and open environment design. CREW provides built-in tasks for rapid development and allows users to integrate customized tasks to accommodate the limitless applications of Human-AI teaming.

Real-time communication. While some Human-AI interaction tasks, such as human preference-based fine-tuning, can be performed offline, many applications require online real-time interaction. Whether it is training decision-making models with real-time human guidance or general human-AI collaboration tasks, the ability to convey messages with minimum delay is essential. Synchronizing data flow between human interfaces, AI algorithms, and simulation engines necessitates the establishment of a real-time communication channel.

Hybrid Human-AI teaming support. Teaming is an essential aspect of our daily jobs. Our vision extends this concept to Human-AI teaming, where both humans and AI operate in teams. Increasing interest in the organization, dynamics, workflow, and trust in multi-human and multi-AI teams highlights the need for a platform capable of distributing and synchronizing tasks, states, and interactions across multiple environment instances and even across physical locations.

Parallel sessions support. A key bottleneck for human-involved AI research is the requirement to conduct experiments with dozens or hundreds of human subjects to obtain trustworthy and reliable conclusions. Such a process can be tedious and time-consuming. To enhance efficiency and scalability, CREW supports multiple independent parallel sessions of the same setting, unconstrained by geographical locations, to obtain the “crowd-sourcing” effects of large-scale experiments. This capability enables experimenters to collectively share experimental data and results.



Comprehensive human data collection. Though human plays an important role in Human-AI teaming, our understanding of human behaviors remains limited and under-explored in existing studies. Therefore, CREW offers interfaces to simultaneously collect multi-modal human data, ranging from active instructions and feedback to passive physiological signals.

ML community-friendly algorithm design. The choice of programming language and libraries should align with the customs and preferences of the ML community. The system design should be modular to allow seamless transitions between tasks and algorithms.





 §.§ Environments



Tasks We select Unity as the simulation engine for CREW due to its popularity in game design and AI research to allow extensible and open environment design. We have implemented four challenging tasks as examples. Multi-player tasks are designed for multi-agent and multi-human teaming research, and single-player tasks are designed for human-guided AI agent learning studies. For each task, we provide both visual and structured state input options. The detailed settings are summarized in Tab. <ref>.

Bowling is a modified version of Atari bowling where each round consists of 10 rolls and the score for each roll is the number of pins hit. Bowling is designed to have the simplest dynamics among our tasks to serve as a rapid test for training a single agent. Find Treasure (Fig. <ref>A) is a single-player embodied navigation task where the agent's goal is to explore a maze and reach the treasure with randomized initial and goal locations. 1v1 Hide-and-Seek <cit.> (Fig. <ref>B) is a one-on-one hide-and-seek task where the seeker learns to explore the maze and catch a moving hider that follows a heuristic policy for obstacle avoidance, and run away from the seeker within line of sight. We introduce this task as an adversarial competition setting. NvN Hide-and-Seek (Fig. <ref>C) is a multiplayer version where multiple seekers and hiders can coordinate, collaborate, and compete. The hiders and seekers can either be controlled by humans or heterogeneous AI policies.






Visual Observations Different visual observations create various challenges in visual embodiment learning for both humans <cit.> and AI agents <cit.>. We provide various camera configurations to support perceptual-motor research. As shown in Fig <ref>, for all our navigation and competitive tasks, we implemented visual observations from multiple views for the users' selection: a top-down fully observable view, a top-down accumulated partially observable view similar to SLAM in robotics <cit.>, a top-down egocentric view, and a first-person view.



Procedural Generation Learning robust, generalizable, and scalable AI agents requires diverse training environments. Procedural generation allows for the creation of a wide range of environment patterns and terrain types. We provide randomized maze patterns where the grids are guaranteed to be connected (Fig. <ref>A) and procedural-generated terrains for more complex simulations as in Fig. <ref>B.





 §.§ Human and Agent Role Assignment



Humans and AI agents often have complementary strengths. For example, humans are generally better at exploring and adapting to new situations, while AI agents are good at repetitive exploitation and precise calculation. Naturally, a team consists of humans and AI agents should have various roles to be effective. Different roles can also be assigned within AI agents to study multi-agent machine learning with heterogeneous policies. To facilitate these experiments, we provide the role assignment feature in CREW.

“Player” allows humans to directly control an agent. “Viewer” allows humans to observe and provide feedback signals as guidance to an AI agent. “Server” role allows humans to host the entire task by monitoring it without direct participation. “AI Agent” assigns different learning policies to each agent.

[13]r0.25


  

  
    < g r a p h i c s >

  

  

  (A) Discrete scalar feedback. (B) Option to take control of the agent and teleoperate. (C) Continuous scalar feedback: the human can hover the mouse over this window to provide per-step feedback.
  


Human Feedback Interface We provide a user interface to allow the Viewer role to provide direct feedback to AI agents shown in Fig. <ref>. Scalar feedback is the most common feedback type used in human-guided RL <cit.>. Conventionally, a human chooses to provide a positive or negative rating to the state-action pair of an agent policy. Additionally, CREW offers an interface that allows humans to provide feedback that is continuous in value and time, enabling granular feedback information. Moreover, our interface also allows humans to directly take control over agents and perform teleoperation.




 §.§ Multiplayer and Parallel Sessions



Enabling multi-human multi-agent sessions requires robust networking solutions (Fig. <ref>). We use Unity Netcode <cit.> for game state synchronization, and Nakama <cit.> as the networking server. In CREW, a server instance hosts the task, runs the simulation, and handles agent policy training, which can be executed on a powerful headless GPU server. Human participants can connect via client instances on less powerful machines, which display synchronized game states and collect human input. CREW is cross-platform, allowing participation from Linux, Windows or MacOS machines.







 §.§ Human and Agent Data Collection



Data collection is at the core of Human-AI teaming research. CREW includes a pipeline for thorough data collection on both the human side and AI agent side. 

Human data Besides the feedback interfaces that collect feedback signals of multiple modalities and teleoperation actions, we also provide interfaces for collecting audio, eye gaze, pupillometry, electroencephalogram (EEG), and electrocardiogram (ECG) physiological responses as in Fig. <ref>.

Agent data including the policy weights, observations, actions, rewards, feedback received, and loss values at every time step can all be saved for further analysis. Users also have the option to enable experiment monitoring and logging by Weights & Biases <cit.>. As all of our tasks include vision-based settings, we also provide implementations of a set of vision encoder architectures.






 §.§ Designing Algorithms



Algorithms research is crucial for Human-AI teaming. We designed the algorithm component of CREW to be extensible and accessible to the ML community. Algorithms are implemented in Python with PyTorch <cit.> and TorchRL <cit.> which is part of the PyTorch ecosystem, making it easy for researchers to design and deploy new algorithms. We provide implementations of state-of-the-art human-guided machine learning algorithms and strong reinforcement learning baselines. The API for communication between a Unity instance and Python algorithm is implemented with MLAgents <cit.>. All environments follow a uniform communication protocol for the observation and action data across tasks. Our modular design allows smooth switching between tasks and algorithms. Real human experiments are time-consuming and costly. To ease algorithm debugging, we implemented simulated human feedback providers as surrogates for real humans for complex tasks. These simulated feedback uses heuristics based on prior task knowledge and is not available in novel tasks in the real world; hence, they should be used solely as debugging tools, not as replacements for real human evaluations.

[16]r0.4


  

  
    < g r a p h i c s >

  

  

  Modular pipeline for experiment setup. Experimenters can freely select and organize the order of cognitive tests and Human-AI experiments with CREW's deployment pipeline.
  






 §.§ Modular Pipeline Design for Quantifying Human Characteristics



Individual differences among humans can significantly affect their teaming with AI agents. To support research along this line, CREW supports a set of cognitive tests to quantify these differences shown in Tab. <ref>. We provide a modular and convenient pipeline (Fig. <ref>) for executing cognitive tests and Human-AI experiments. The framework integrates various media files (e.g., instruction videos or pictures), inter-trial intervals, executable Python scripts, and Unity builds, ensuring a smooth and effective workflow. The pipeline requires minimal effort from researchers during proctoring, as a single click initiates the sequential execution of experiments. The pipeline allows restart from interruption points.








§ BENCHMARKING STUDY




As an example of running experiments with CREW, we benchmark a state-of-the-art real-time human-guided RL framework, Deep TAMER <cit.>, along with strong RL baselines. In the original Deep TAMER, the framework was only tested on  Atari Bowling with 9 human subjects. With CREW, this is the first time it is possible to systematically conduct human-guided RL benchmarking across multiple environments on a larger population. We summarize our findings as well as insights on the scalability of the framework in this section. We also discuss the relationship between human characteristics and guided agent performance.







 §.§ Experiment Setup
 


Tasks We selected 3 single-player games: Bowling, Find Treasure, and 1v1 Hide-and-Seek for this benchmark. For Find Treasure and Hide-and-Seek, each episode has a time limit of 15 seconds. All algorithms directly learn from visual inputs with the top-down accumulated partially observable view. 

Human Trainers We recruited 50 human subjects for the experiments under the approval of the Institutional Review Board. For each 1.5-hour session, the subject will receive $20 compensation. Benefiting from CREW's unique feature to host experiments with parallel sessions, we were able to conduct all 50 experiments within one week. Prior to guiding the agents, the participants were asked to complete all our cognitive tests in Tab. <ref>.


Deep TAMER leverages human feedback as value functions. During training, a human trainer provides positive or negative discrete feedback based on the agent's behavior. This feedback is assigned to relevant state-action pairs through a credit assignment window. A neural network is trained to predict human feedback given state-action pairs, and the policy chooses actions that maximize this predicted feedback. Originally, Deep TAMER was limited to discrete actions. We extend it to continuous action spaces using an actor-critic framework, termed as c-Deep TAMER. For bowling, the agent is trained for 5 minutes, and for 10 minutes in Find Treasure and Hide-and-Seek.

Baseline RL We selected two state-of-the-art RL algorithms: Deep Deterministic Policy Gradient (DDPG) <cit.> and Soft Actor-Critic (SAC) <cit.>. 

Heuristic feedback We also evaluate simulated feedback-guided RL. We simply add the feedback signals as additional dense rewards to the environment reward. DDPG is selected as the backbone RL algorithm as the state-of-the-art transitioned from SAC to DDPG <cit.> in visual control tasks.

Evaluation We checkpoint model weights every 2 minutes and evaluate on unseen test environments. For bowling, every checkpoint is evaluated for 1 game (10 rolls). For Find Treasure and 1v1 Hide-and-Seek, the checkpoints are evaluated for 100 episodes.




 §.§ Results



Agent training performance We hypothesize that subjects with higher cognitive tests can lead to higher-performing agents. Therefore, we show the agent performances guided by the 15 subjects who scored the highest in our cognitive tests side by side with the performance of all 50 subjects in Fig. <ref>. As shown in the results, the agents guided by the top 15 subjects exhibit higher performance than the overall average. In particular for the top 15 subjects, on the simple bowling task, c-Deep TAMER surpasses RL baselines by an average of 10 scores given the same training time. On Find Treasure, heuristic feedback achieved the highest performance as expected, showing the upper bound performance with accurate and non-delayed dense rewards. c-Deep TAMER also demonstrated strong performance with faster learning trends than RL baselines. On 1v1 Hide-and-Seek, c-Deep TAMER performed similarly to RL baselines, suggesting that c-Deep TAMER has difficulty scaling to tasks with higher complexity. Similar conclusions still hold for all 50 subjects.













Analysis of Individual Differences Due to the cognitive test feature and emphasis on Human-AI teaming, we can deepen our understanding of how individual human differences can affect the performance of human-guided agents. We calculated the correlation between human subjects' cognitive test scores and c-Deep TAMER training results in Fig. <ref>. The cognitive test scores are normalized by the mean and variance over the subjects through z-score, and outliers with scores 1.5× Interquartile Range above the third quartile or below the first quartile are removed. We found that non-outlier subjects performed equally well on the spatial mapping test so we do not include this for correlation analysis. The heatmap shows the sign of correlation coefficients from linear regression and the statistical significance (*).


[17]r0.5


  

  
    < g r a p h i c s >

  

  

  Correlation between cognitive test scores and c-Deep TAMER training performance. The darker the color, the more statistically significant the correlation. “+” or “-” means positive or negative correlation.
  

The Find Treasure score has a significant positive correlation with theory of behavior, mental rotation, and fitness scores, suggesting that subjects who performed better on these tests also trained better agents. This is likely due to the need for good spatial reasoning and future behavior estimation in Find Treasure. The 1v1 Hide-and-Seek score shows a significant positive correlation with reflex score, likely due to the rapid changes in environment and task dynamics in the complex adversarial setting. Subjects with faster reaction speeds provided timely feedback to align with relevant state-action pairs. Overall, the total score (i.e., the sum of three game scores) has significant positive correlations with rotation, fitness, and overall score, suggesting that these cognitive skills are most relevant to the performance of c-Deep TAMER-guided agents.




§ CONCLUSION, LIMITATION, AND FUTURE WORK



We introduce CREW for facilitating Human-AI teaming research from diverse human and machine learning scientific communities. CREW offers extensible environment design, enables real-time human-AI communication, supports hybrid Human-AI teaming, parallel sessions, multimodal feedback, and physiological data collection, and features ML community-friendly algorithm design. We also provide a set of built-in tasks and baseline algorithm implementations. Using CREW, we benchmarked a state-of-the-art human-guided RL algorithm against baseline methods involving 50 human subjects and provided insights into the relationship between individual human differences and agent-guiding performance.

CREW is still in the early efforts among several critical aspects. Future work will explore building more diverse and challenging tasks, including multiplayer tasks with complex strategies and robotics environments requiring an understanding of physics. While we have only benchmarked several algorithms, we hope CREW can help benchmark many existing algorithms that were not fully open-sourced in a unified environment. Finally, more supports on human physiological data processing and analysis shall be investigated and supported in CREW.



§ ACKNOWLEDGEMENTS

We thank the members at Duke General Robotics Lab, Sida Zhu and Harish Yerra for early development on the platform, and Jiaxun Liu and Easop Lee for beta testing. This work is supported in part by ARL STRONG program under awards W911NF2320182 and W911NF2220113.













































































































    

















§ ACCESSING CREW

CREW is a platform for cross-disciplinary human-AI teaming research. The codebase for our platform, including environments, interfaces, and algorithms can be accessed through <https://github.com/generalroboticslab/CREW>. Videos, documentation, and tutorial are hosted on <http://generalroboticslab.com/CREW>. Our platform is fully open-sourced for academic use.








§ PLATFORM DETAILS

The environments of CREW is implemented using , with packages  <cit.>,  <cit.> and  <cit.>. Algorithms are developed with  <cit.>.




§ ADDITIONAL RESULTS




 §.§ Feedback Visualization

In Fig <ref> we show examples of the state-action pairs and human assigned feedback value during c-Deep TAMER training. Since humans trainers do not see the action (next navigation destination) explicitly, we show consecutive frames instead.





 §.§ Full Cognitive Test Analysis

We include the full results for the cognitive test and c-Deep TAMER score analysis. All linear regression plots including coefficients and p-values and summarized in Fig <ref>.









§ COMPUTATIONAL RESOURCES


All human subject experiments were conducted on desktops with one NVIDIA RTX 4080 GPU. All evaluations were run on a headless server with 8 × NVIDIA RTX A6000 and NVIDIA RTX 3090 Ti.





§ BENCHMARKING EXPERIMENT DETAILS



 §.§ Implementation details


c-Deep TAMER Algorithm
The detailed c-Deep TAMER algorithm is summarized in Alg <ref>. It is modified from the original Deep TAMER integrated with an actor network for action selection. The actor is updated by gradient ascent that outputs the actions that maximizes the predicted human feedback.


Processing Inputs All tasks in our experiments directly learn from pixels. The frames rendered from the environment are first resized to 100×100 pixels. For Find Treasure and 1v1 Hide-and-Seek, we stack the past 3 frames as input. We then apply simple random shift to the frame stack, as it has been shown to be an effective augmentation strategy for visual reinforcement learning. The shifting factors along the height and width are uniformly sampled from [0, 0.08].


Model architecture We use a 3-layer CNN as the vision encoder, each having 64 channels and followed by batch normalization and ReLU activation function. All actor-critic frameworks uses a 3-layer MLP with 256 neurons in each layer for both the actor and critic network. 



 §.§ Hyperparameters

The hyperparameter settings for out experiments is summarized in Tab <ref>.






 §.§ Human subject experiment details


Overview
We recruit the human subject on campus by using flyers and emails. For every human subject, the experiment time is approximately 40 minutes without interruptions. The experiment starts with cognitive tests (10 minutes)  and is followed by the human guiding the agent using the c-Deep TAMER framework (30 minutes). The order of the cognitive tests is Eye Alignment, Reflex, Theory of Behavior, Mental Rotation, Mental Fitting, and Spatial Mapping. There are detailed instruction videos for each test before the test starts. As for the human guiding agent part, each human subject guides the agent to play 3 games for a total of 30 minutes(5 minutes for Bowling, 10 minutes for Find Treasure, and 10 minutes for 1v1 Hide-and-Seek). Before each game, there is a detailed instruction video that describes the rule of the game and how human subjects can give feedback to the agent, which we included below.



  §.§.§ Instruction Video Script


 General Introduction:  
 
 

Cognitive Test Introduction: 



Eye Alignment Instruction: 



Reflex Instruction: 



Theory of Behavior Instruction: 



Mention Rotation and Mental Fitting Instruction: 



Spatial Mapping Instruction: 



Human Guiding AI Introduction: 



Bowling Instruction: 



Find Treasure Instruction: 



1v1 Hide-and-Seek Instruction: 






  §.§.§ Compensation

We pay each human subject $20 for participation. 



§ AUTHOR STATEMENT

The authors bear all responsibility in case of violation of rights.




































 Paper Title: Who should I trust? A Visual Analytics Approach for Comparing Net Load Forecasting Models.
































Who should I trust?A Visual Analytics Approach for Comparing Net Load Forecasting Models



Parts of this work were supported by the U.S. Department of Energy Solar Energy Technologies Office and the Office of Electricity Sensors Program, and performed jointly at the Pacific Northwest National Laboratory under Contract DE-AC05-76RL01830 and at the Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344).

    Kaustav Bhattacharjee1, Soumya Kundu2, Indrasis Chakraborty3, and Aritra Dasgupta1

1Department of Data Science, New Jersey Institute of Technology, USA. Email: {kb526, aritra.dasgupta}@njit.edu 
2Optimization and Control Group, Pacific Northwest National Laboratory, USA. Email: soumya.kundu@pnnl.gov
3Center for Applied Scientific Computing, Lawrence Livermore National Laboratory, USA. Email: chakraborty3@llnl.gov 

    August 5, 2024
========================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Net load forecasting is crucial for energy planning and facilitating informed decision-making regarding trade and load distributions. However, evaluating forecasting models' performance against benchmark models remains challenging, thereby impeding experts' trust in the model's performance. 

In this context, there is a demand for technological interventions that allow scientists to compare models across various timeframes and solar penetration levels. 

This paper introduces a visual analytics-based application designed to compare the performance of deep-learning-based net load forecasting models with other models for probabilistic net load forecasting. 

This application employs carefully selected visual analytic interventions, enabling users to discern differences in model performance across different solar penetration levels, dataset resolutions, and hours of the day over multiple months. We also present observations made using our application through a case study, demonstrating the effectiveness of visualizations in aiding scientists in making informed decisions and enhancing trust in net load forecasting models.



visual analytics, AI/ML, net load forecasting




§ INTRODUCTION



Comparing multiple computational models for their performance is crucial for enhancing trust in model outcomes because it provides a basis for evaluating the reliability and consistency of each model <cit.>. By assessing how different models perform under various conditions and scenarios, stakeholders can better understand their strengths, weaknesses, and overall effectiveness. Such comparative analysis helps to identify the most suitable model for specific tasks or applications, thereby instilling confidence in the reliability of the chosen model. This becomes more important while predicting the net load of an electric grid, which is defined as the difference between total electricity demand and generation from behind-the-meter resources like solar and distributed generators, and is influenced by various factors such as weather conditions and time of day <cit.>. 

Accurate net load forecasting enables grid operators, policymakers, and energy providers to make informed decisions regarding energy trade, load distribution, and resource allocation. However, the rise of solar energy generation sources in residential settings has significantly impacted the performance of traditional net load forecasting models, highlighting the need for robust time-series forecasting techniques <cit.>. Collaborating with scientists, we developed a deep-learning model that integrates variables such as temperature, humidity, apparent power, and solar irradiance to achieve strong predictive performance and resilience in the face of missing data <cit.>. However, the model's sensitivity to seasonal variations and noisy inputs underscores the need for further exploration with domain experts. Hence, we developed an interactive tool that empowers users to investigate the model's performance across diverse time periods and input scenarios <cit.>.





However, while we initially recognized the importance of comparing the model's performance with traditional models, feedback from domain experts further emphasized its significance in building trust in model outcomes. Visual analytics can play a pivotal role here, as evidenced by prior research demonstrating its importance in enhancing trust in machine learning models <cit.>.

This is exemplified by the interactive tool we developed, which enabled domain experts to extract valuable insights concerning the model's sensitivity toward temperature and humidity. Moreover, recent discourse, as highlighted in <cit.>, emphasizes the critical role of visual analytics in fostering trust-augmented applications of artificial intelligence and machine learning (AI/ML) within the energy sector. Building upon this, we designed  , incorporating carefully selected visual analytic interventions. These interventions facilitate the comparison of multiple models across various parameters, including solar penetration levels, dataset resolutions, and different hours of the day, enhancing stakeholders' confidence in model performance.



The aim of our application is to build trust through model comparison, primarily comparing our net load forecasting model with a reference model. We enhanced our model to process inputs at varying resolutions and then devised the reference model, which generates predictions by averaging net load ground truths for the last 30 days at the same time point. Despite lacking predictive ML components, this reference model serves as a benchmark in various net load forecasting competitions, including the recent Net Load Forecasting Prize by the National Renewable Energy Laboratory (NREL) and the U.S. Department of Energy Solar Technologies Office (SETO) <cit.>. Through the web interface of , we were able to uncover patterns in the performance that help improve trust in the model outcomes. In this work, we identify the visual analytic tasks that are required to compare these models. These tasks can be extended to compare other similar net load forecasting models, enabling users to make well-informed decisions based on the outcomes of these models.

While previous research has predominantly focused on developing interfaces during the model development phase, tailored to assist model developers, our emphasis lies in the post-hoc evaluation of model performance, specifically addressing the needs of energy scientists and grid operators <cit.>. Other studies have explored the performance of probabilistic net load forecasting models through different visualization charts but have often lacked an integrated interactive interface <cit.>. In contrast, our interactive visual analytic tool provides carefully designed visual cues for comparing model performance across different factors like dataset resolutions and solar penetration levels, thus offering a novel approach in this domain.

In this paper, we first introduce the different models used in this work and the rationale behind choosing them. Subsequently, we outline the identified visual analytic tasks and detail the design decisions guiding the development of our interactive interface. This is followed by some of the observations made by our power scientist collaborator through that demonstrate its efficacy in comparing model performance across multiple facets. Finally, we conclude by sharing insights gained from this development and discussing some of the future research opportunities in this domain. Additionally, a short demonstration video is available https://bit.ly/trustvishere.



§ MODEL DESCRIPTION




We start with a deep learning-based probabilistic model tailored for net load forecasting in high behind-the-meter solar scenarios <cit.>. This model has three key components: a kernelized probabilistic forecasting (kPF) module, an autoencoder (AE), and a long short-term memory (LSTM) network. This model effectively captured complex temporal dependencies and uncertainties inherent in net load data, which is crucial for reliable forecasting in environments with high solar penetration. By incorporating kernel methods into probabilistic forecasting, the model handled non-linear relationships and captured subtle variations in net load influenced by solar energy fluctuations. The autoencoder component enhanced feature extraction and dimensionality reduction, facilitating the LSTM network's ability to capture long-term dependencies and predict future net load values accurately. Experimental results demonstrated the superior performance of this model compared to traditional forecasting models, showcasing its efficacy in addressing the challenges posed by high solar scenarios and advancing the state-of-the-art in net-load forecasting methodologies. 

This model was used in the Net Load Forecasting Prize competition hosted by NREL and SETO <cit.>. However, during this competition, we observed its underperformance on the data provided by the organizers, which had lower resolutions. We discovered that while the autoencoder component excelled with high-resolution datasets (such as 15-minute intervals), it struggled to effectively capture temporal dependencies in lower-resolution datasets (e.g., 1-hour intervals provided by the organizers). Consequently, this limitation led to subpar outcomes generated by the LSTM component. In light of this, we opted to remove the kPF and autoencoder components and instead developed a version of the model solely utilizing the LSTM component. Additionally, fine-tuning the number of layers in the LSTM component yielded significantly improved results in the competition.

The competition employed a reference model to assess model performance across various probability levels. As previously mentioned, this reference model simply utilizes historical input data from the past 30 days to generate probabilistic forecasts for a specific time point. This model can serve as the initial benchmark for assessing the effectiveness of other models. Therefore, in this work, we developed a reference model following the same principles. Since these forecasts are probabilistic in nature, we calculated the Continuous Ranked Probability Score (CRPS) for both the reference model and our model. Subsequently, we computed the Continuous Ranked Probability Skill Score (CRPSS) based on these CRPS scores, evaluating whether our forecast presents an improvement or deterioration compared to the reference forecast <cit.>. A positive CRPSS indicates that the forecast outperforms the reference forecast, whereas a negative value suggests inferior performance. Utilizing these CRPSS values, we compare the performance of our model against the reference model across multiple dates throughout the year, and present these values in . 



§ VISUAL ANALYTICS-BASED DESIGN


Our application employs a visual analytics-based design featuring coordinated views enhanced with visual cues to help users compare model performance. It combines interactive visualization with comparison metrics like CRPSS to improve trust in the model outcomes and also allows the users to probe the model and understand its performance across different solar penetration levels and months. In this section, we highlight the two tasks performed by and how its visual analytics-based design aids in executing these tasks:
T1: Compare model performance across different solar penetration levels and data resolutions: Model performance may vary across different solar penetration levels due to the increased variability in net load data caused by intermittent solar generation. LSTM-based models might struggle to capture and predict these dynamic behaviors accurately. Furthermore, datasets with higher resolutions, such as sub-hourly intervals, enable models to more effectively capture short-term fluctuations and dependencies. Hence, this task essentially involves comparing model performance at different solar penetration levels and data resolutions.
T2: Identify patterns across different timeframes: By analyzing performance across multiple timeframes, power scientists can assess the net load forecasting models' robustness and consistency in capturing both short-term fluctuations and long-term trends. This evaluation helps to identify whether a model's performance is consistent across various temporal scales or exhibits variability or biases at specific time periods. This task relates to identifying patterns in the model's performance across different months, various hours of the day, and different time periods.

Our application implements multiple coordinated views and components in order to fulfill these tasks. Next, we discuss the design of these views and components along with the rationale behind them:
Comparison View: As comparing model performance is the main objective of , we begin with the Comparison View, which utilizes modified box plots to compare models. Figure <ref>a depicts CRPSS values on the y-axis and different data resolutions on the x-axis. The box plots illustrate the distribution of CRPSS values, with the median typically exceeding zero, indicating superior performance of our model over the reference in most cases. Users can utilize the solar penetration level filter to compare performance across various levels (20%, 30%, 50%) (Figure <ref>c) (T1). However, based on initial feedback from domain experts, we recognized the importance of displaying the distribution of CRPSS values. Therefore, we integrated dots representing the CRPSS values, jittered along the x-axis to illustrate their distribution over the entire year. These dots align with insights from the box plot and additionally reveal instances where our model performs worse than the reference on certain dates. This led to the development of the Patterns View, where we can identify the timeframes where the model underperforms compared to the reference.

Patterns View: In this view, utilizes a heatmap to depict performance patterns for each month across different hours of the day (T2). The x-axis represents the 24 hours of the day (0-23), while the y-axis displays the months of the year (Feb - Dec) (Figure <ref>b). Each box in the heatmap denotes the average CRPSS value for each month at each hour, indicated by the color. Darker blues signify more positive CRPSS values, indicating the superior performance of our model compared to the reference at that time. Conversely, darker reds indicate more negative CRPSS values, signifying poorer performance of our model compared to the reference at that time. This diverging color scale aids in easily identifying performance patterns across different time points and facilitates the identification of instances where our model does not outperform the reference. Users can filter specific months to focus on particular time periods and analyze performance accordingly (Figure <ref>e). Additionally, based on feedback from domain experts, we implemented the Sidebar component to allow users to select specific date ranges and analyze performance patterns within those ranges.
Sidebar: The Sidebar facilitates user selection of start and end dates to filter results across all views, enabling focus on specific date ranges (T2) for comparing model performance within those periods (Figure <ref>d). Additionally, based on feedback from domain experts expressing interest in comparing model performance across all solar penetration levels simultaneously, offers a comparison mode toggle in the Sidebar. Enabling this mode updates both views to display box plots and heatmaps for all solar penetration levels side by side, aiding in point-to-point comparison and identification of performance patterns across different solar penetration levels.



§ RESULTS FROM A CASE STUDY

The efficacy of an application can be validated if the application is able to perform the intended tasks effectively. In this section, we show the results through a case study that demonstrates how can be used to compare and select net load forecasting models effectively.

This case study involved a power scientist with over 10 years of experience in power and grid systems. With expertise in nonlinear dynamics, large-scale networks, and distributed control, he played a crucial role in developing the model. His main objective was to assess the model's performance relative to the reference model across various time points and solar penetration levels, which are critical factors to consider before deploying it for a project. We informed him that we had integrated CRPSS values for both the model and the reference model for all dates throughout the year. He then accessed through a browser and examined the distribution of CRPSS values across different data frequencies at a 20% solar penetration level (Figure <ref>a). Notably, he discovered that the model performed better with the lower resolution dataset (1-hour), contrary to expectations for an LSTM-based model (T1). Upon further examination, he noted a marginal difference in the median CRPSS between high and low-resolution datasets (0.85 and 0.89, respectively). Therefore, he enabled the comparison mode through the Sidebar, enabling him to assess the model's performance as solar penetration increased. He noted that the model consistently performed well with the higher resolution dataset (15-min) across all other solar penetration levels, confirming the notion that our model excels with high-resolution datasets in high solar penetration scenarios (Figures <ref>b and  <ref>c). Next, he observed that although the median CRPSS was significantly above zero, the minimum value was negative. Upon inspecting the dots adjacent to the box plots in the Comparison View, he observed that while most dots clustered around the median line, there were a few outliers with negative CRPSS values.

Hence, the scientist sought to understand temporal patterns to identify if any specific hour of the day contributed to the negative values. Consequently, he navigated to the Patterns View within to analyze the CRPSS value distribution in the heatmap at a 20% solar penetration level (Figure <ref>d). Consistent with observations from the Comparison View, most heatmap boxes displayed varying shades of blue, indicating superior model performance compared to the reference across most months and hours. On closer look, the scientist noted a box-like pattern in the heatmap, revealing enhanced performance during morning hours (8 am to 10 am) from April to September, followed by a decline during midday (11 am to 4 pm), and another spike in the evening (4 pm to 6 pm) during these months. This pattern suggested that the model effectively captured diurnal variations in net load data and adjusted predictions accordingly (T2). While similar box-like patterns were evident across other solar penetration levels, the intensity of model performance varied with increased solar penetration. This insight holds practical significance for model selection during deployment, as it suggests the potential use of different models or model ensembles tailored to distinct times of the day, leveraging their respective performances on diurnal patterns. Thus, the power scientist was satisfied that could yield valuable insights regarding the model, aiding informed decision-making. A short demonstration video is available at https://bit.ly/trustvisthis link.


§ CONCLUSION

Efficient model selection for net load forecasting plays a pivotal role in energy planning and grid operations. In this work, we delve into this process, integrating visual analytics with input from domain experts to identify key tasks for comparative model selection. Subsequently, we translate these tasks into an interactive interface, enabling users to assess model behavior across various factors such as solar penetration levels, data resolution, and time of day.


Throughout this endeavor, we gained valuable insights into the model behavior and the challenges posed during the multi-way comparison of models. This collaborative effort underscored the need for interactive tools that facilitate the seamless translation of model insights into actionable decisions. We can argue that our application is a first step towards this direction. As a next step, our plan is to incorporate multiple net load forecasting models into the application and integrate additional metrics for effective comparison of their performance.

Looking ahead, we aim to enhance our application in several ways. Incorporating economic planning and analysis options will allow stakeholders to assess the cost-benefit ratio before model selection, thereby enhancing trust in the outcomes. Additionally, as the energy landscape evolves, our application's flexibility will be pivotal in effectively comparing and selecting forecasting models for real-world applications. In summary, our collaborative endeavor has produced a robust tool for multi-faceted model comparison and has paved the way for informed decision-making through visual analytics in energy planning.








abbrv-doi




 Paper Title: Can Developers Prompt? A Controlled Experiment for Code Documentation Generation.














Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
    

Hans-Alexander Kruse 0009-0006-5568-7841,
Tim Puhlfürß 0000-0001-8421-8071,
Walid Maalej 0000-0002-6899-4393


Universität Hamburg
  
Hamburg, Germany

hans-alexander.kruse@studium.uni-hamburg.de,
tim.puhlfuerss@uni-hamburg.de,
walid.maalej@uni-hamburg.de

    August 5, 2024
======================================================================================================================================================================================================================================================================



§ ABSTRACT 

llm bear great potential for automating tedious development tasks such as creating and maintaining code documentation.

However, it is unclear to what extent developers can effectively prompt llm to create concise and useful documentation.

We report on a controlled experiment with 20 professionals and 30 computer science students tasked with code documentation generation for two Python functions.
The experimental group freely entered ad-hoc prompts in a ChatGPT-like extension of Visual Studio Code, while the control group executed a predefined few-shot prompt.

Our results reveal that professionals and students were unaware of or unable to apply prompt engineering techniques.
Especially students perceived the documentation produced from ad-hoc prompts as significantly less readable, less concise, and less helpful than documentation from prepared prompts.
Some professionals produced higher quality documentation by just including the keyword Docstring in their ad-hoc prompts.
While students desired more support in formulating prompts, professionals appreciated the flexibility of ad-hoc prompting.
Participants in both groups rarely assessed the output as perfect.
Instead, they understood the tools as support to iteratively refine the documentation.

Further research is needed to understand which prompting skills and preferences developers have and which support they need for certain tasks.



Software Documentation,
Large Language Model,
Program Comprehension,
Developer Study,
AI4SE


Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
    

Hans-Alexander Kruse 0009-0006-5568-7841,
Tim Puhlfürß 0000-0001-8421-8071,
Walid Maalej 0000-0002-6899-4393


Universität Hamburg
  
Hamburg, Germany

hans-alexander.kruse@studium.uni-hamburg.de,
tim.puhlfuerss@uni-hamburg.de,
walid.maalej@uni-hamburg.de

    August 5, 2024
======================================================================================================================================================================================================================================================================










§ INTRODUCTION




Developers often overlook or ignore software documentation and generally assign it a low priority <cit.>. 
Yet, carefully documenting code is an essential task in software engineering. 
Up-to-date and high-quality documentation facilitates program comprehension <cit.>, accelerates developer onboarding <cit.>, and mitigates technical debt <cit.>. 
Software documentation is also central to software maintenance, as documentation often requires updates and should evolve with software  <cit.>.


Researchers and tool vendors have thus investigated different ways to automate documentation <cit.>. 
In particular, llm designed to model and generate human language <cit.> hold great potential in automating documentation tasks <cit.>.
Among popular llm use cases, code summarization has recently gained much attention <cit.>.
However, the output of llm usually depends on the user input, called prompt. 
Slightly changing the prompt can lead to different results regarding conciseness, language style, and content of generated text <cit.>. 


Several prompt engineering techniques have emerged to optimize interactions with llm <cit.>.
Few-shot prompting is one of the most prominent techniques and comprises adding example outputs to a predefined prompt to specify output requirements <cit.>.
This technique allows users to optimize llm responses while minimizing the number of input messages sent to the model <cit.>.


Previous research has primarily focused on benchmarking the performance of various models and prompt engineering techniques for generating source code documentation <cit.>.
Recently, Ahmed and Devanbu concluded that common prompt engineering techniques perform better than ad-hoc prompting according to popular benchmark metrics <cit.>.
However, studies focusing on the perspectives of developers when using llm, e.g., to generate documentation, are still sparse.
In fact, a recent study showed that common metrics to evaluate llm do not align with human evaluation to assess the quality of generated documentation <cit.>.
Hence, it remains uncertain whether prompt engineering techniques meet developers' requirements for generating and using code documentation.
Moreover, it is unclear whether developers prefer a flexible, iterative interaction using chatbot-like ad-hoc prompting or a rather transaction-like execution of predefined prompts.

This study takes a first step towards filling this gap, focusing on two research questions:



  RQ1: How well can developers prompt llm to generate code documentation compared to a predefined prompt?

  RQ2: How is the developer experience for ad-hoc prompting compared to executing predefined prompts?



To answer these questions, we conducted a randomized controlled experiment with 20 professional developers and 30 computer science students.
The experiment involved generating code documentation using an llm-powered ide.
The first group utilized a vscode extension based on GPT-4, enabling participants to enter ad-hoc prompts for selected code.
The second group used a similar vscode extension that executed a predefined few-shot prompt to generate code documentation via GPT-4 with a single click.



Overall, we observed that developers with less experience require more guidance in phrasing an ad-hoc prompt, whereas more experienced developers can generate fairly good documentation by including specific keywords like Docstring in the prompts.
Overall, participants preferred transactional or guided llm interactions to create code documentation while enjoying the flexibility of ad-hoc and iterative prompting.
Our results show that predefined prompts deliver code documentation of significantly higher quality than ad-hoc prompts, especially due to a consistent documentation format.

We present the design of our study in Section <ref> and report on the results in Section <ref> (RQ1) and Section <ref> (RQ2). 
We then discuss the implication of our findings in Section <ref> and potential threats to validity in Section <ref>. 
Finally, Section <ref> summarizes related work, and Section <ref> concludes the paper.
We share our experiment results, including the ide extension, in our replication package <cit.>.







§ METHODOLOGY





 §.§ Experimental Design




We designed a controlled experiment in a laboratory setting to compare developer usage of and experience with two vscode extensions employing different prompts for code documentation generation.
We followed a between-subject design, with one participant testing only one of both tools to avoid learning effects <cit.>.
The ad-hoc prompt group created prompts in an already available GPT-4-powered ide extension, while the predefined prompt group interacted with our extension.
We randomly assigned participants to the groups, with adjustments made to balance the occupations. 
Participants were unaware of their group.


We offered offline and online versions of the experiment.
In the offline version, participants used our computer with vscode and the digital questionnaire displayed on two screens.
In the online version, we utilized Zoom for video conferencing, sharing a questionnaire hyperlink, and screen-sharing our vscode window with external input control.
Participants employed Thinking Aloud to express thoughts during the task <cit.>.
To ensure a calm environment, we put a participant in a room empty of people.
The questionnaire included questions about the code and the generated documentation.
Additionally, we collected the ad-hoc prompts for a later comparison with the predefined prompt.


The experiment comprised three parts, as shown in Figure <ref>.
In the first part, participants answered questions about occupation, as well as Python and vscode experience to assess their ability to understand complex functions and use the ide.
Furthermore, we could check whether these variables have any effect on the observed behavior.

The second part focused on code comprehension and documentation generation.
It consisted of two rounds, each involving a pre-selected Python function (Listing <ref>).
Per round, participants (1) first attempted to comprehend the code without any documentation.
This enabled us to analyze the effect of the llm-generated documentation on understanding the code functions.
Subsequently, (2) participants rated their comprehension on a scale from 1 (very low) to 5 (very high), and answered True/False questions about the code to further check their understanding (Table <ref>).
Participants could also state that they had insufficient information to answer a question.
Furthermore, (3) participants manually created a function comment to externalize their understanding and better assess the quality of the comment subsequently generated by the tool.
To guide the participants and shorten the completion time, we provided the participants with a comment template adhering to guidelines for clean Python code <cit.> and instructed them to add only the function description without the explanations of parameters.
Participants then used the respective llm-powered ide extension to generate code documentation.
Afterward, (4) as documentation should serve to facilitate code comprehension, they studied the generated comment and revisited the True/False questions.
This step provided insights if the documentation supported or even hindered the comprehension.
Finally, (5) they rated the generated documentation on six quality dimensions based on the human evaluation metrics by Hu et al. <cit.>.

The documentation quality dimensions each provided a question and a five-point answer scale, addressing grammatical correctness, readability, missing relevant information, unnecessary information presence, usefulness for developers, and helpfulness for code comprehension. 
We made minor adjustments to the original answers of two dimensions as we found them overloaded for our goal.
In particular, the answers for grammatical correctness originally also included a rating of fluency, and the original readability answers also concerned the comment's understandability.
The lowest point of each scale represents the negative pole, like very low readability, while the highest point represents the positive pole, like very low amount of unnecessary information.

In the third part of the experiment, participants assessed the usability of the respective ide extension.
They completed the standardized ueq with 26 items, each associated with a seven-point answer scale and one of six usability categories <cit.>.
Finally, participants provided comments on tool strengths and areas for improvement in two free-text fields.



 §.§ Ethical Considerations



We followed the standard procedure of the ethics committee of our department for assessing and mitigating risks regarding ethics and data processing.
Upon welcoming participants, we explained the privacy policy, study purpose, upcoming tasks, and data processing procedures.
We designed the study to last for approx. 30 minutes per participant to prevent exhaustion and instructed participants not to seek perfect solutions to reduce stress. 
We offered clarifications on tasks and questionnaire content without providing solutions.
We minimized observational bias by facing away while participants answered the questionnaire and also encouraged negative feedback <cit.>.
To express our gratitude for the participation, we raffled vouchers among participants.
We pseudonymized all published data as far as possible while maintaining our study objective.



 §.§ Technical Setup



[float, caption=Python functions that participants had to comprehend and for which they generated documentation., label=lst:exp_fct]list=no
Function 1:
def string_from_vector_bool(data):
    return ",".join(str(int(i)) for i in data)
—————————————————-
Function 2:
def _parse_date(date):
    if date is None:
        date = Timestamp()
    if isinstance(date, Timestamp):
        date = date.toLocal()
    d = dateutil.parser.parse(date)
    if d.tzinfo is None:
        d = d.replace(tzinfo=dateutil.tz.tzlocal())
    return d.astimezone(dateutil.tz.tzutc())
    .replace(tzinfo=None).isoformat()


[float, caption=Predefined few-shot prompt. Few-shots 2 and 3 are comparable to 1 and included in the replication package <cit.>., label=lst:const_prompt, xleftmargin=0pt]list=no
For the following prompt, take into account these 
three input/output pairs of functions and 
corresponding appropriate comments:

Function 1:
def elapsed_tid(cls, reference, new):
    time_difference = new.toTimestamp() - reference.toTimestamp()
    return np.int64(time_difference * 1.0e6 // cls._period)
Comment 1:
"""
    Calculate the elapsed trainId between reference and newest timestamp

    :param reference: the reference timestamp
    :param new: the new timestamp

    :type reference: Timestamp
    :type new: Timestamp

    :returns: elapsed trainId's between reference and new timestamp
"""
Function 2:
...

Generate a comment for the following function: 
FUNCTION_CONTENT. Fill in this template: 
TEMPLATE. Adhere to the appropriate comment 
syntax for multi-line comments.




  §.§.§ Selection of Experiment Tasks



We selected the Python source code for the experiment tasks from the open-source control system Karabo <cit.>.
Karabo is maintained by the scientific facility euxfel, whose developers partially participated in the experiment.
By choosing this project and involving project insiders, we could assess how their programming and domain experience supports their code comprehension and documentation rating, especially in comparison to outsiders.

Two authors conducted a thorough screening of available Karabo functions and identified six candidates.
All six were utility functions, which we deemed comprehensible for experiment participants not associated with euxfel.
Our selection criteria also included code documentation quality aspects such as conciseness, completeness, and usefulness especially relevant for the few-shot examples <cit.>.
We independently analyzed the candidates and categorized them into the complexity levels easy, medium, and hard.
Afterward, we resolved categorization conflicts by discussion.

We selected one easy-, and one medium-rated function for the tasks of the experiment (Listing <ref>).
The first function converts a boolean vector into a comma-separated string.
We considered the code easy to understand as only standard Python operations were used within this short function.
The second function converts a given date to the ISO format.
The complexity of this function lies in the usage of multiple classes and functions with abbreviated identifiers, and the lack of inline comments.
Hence, participants must make assumptions based on identifier names and usage.

Initially, we also selected a third function rated hard but removed it after a pilot study with three software engineering researchers, who reported difficulties with this function and far exceeded the time limit.
While we acknowledge that llm might require more sophisticated prompts and context information to generate appropriate documentation for more complex code, our user study focused on how differently skilled developers can and prefer to interact with an llm for the creative process of documentation generation.
Therefore, within the experiment's time limit, all participants should be able to phrase ad-hoc prompts that generate appropriate comments for these functions.



  §.§.§ LLM Selection and Prompt Construction



Research and industry have introduced multiple llm with individual strengths and limitations. 
For our study, we focused solely on the GPT-4 model by OpenAI (version gpt-4-0613). 
We did not aim to benchmark different models but rather to study developers prompting skills particularly for code documentation generation.
OpenAI's models have been prominent in recent research on documentation generation <cit.>. 
Furthermore, during our study period, GPT-4 demonstrated general superiority over other models <cit.>.

We defined the predefined few-shot prompt (Listing <ref>) based on prompt engineering guidelines <cit.> and tested it in a pilot study.
Given GPT-4's limited context window at the time of the study, we applied a three-shot approach to specify the required output format.
We chose three distinct function-comment pairs from the remaining previously selected Karabo code candidates.
Hence, these functions were from the same domain and of similar complexity as the two used in the experiment.
They adhered to a structured documentation format with precise and concise content.
We instructed GPT-4 to generate a comment for the selected source code that we passed as input (FUNCTION_CONTENT).
To further enforce the clean code format <cit.>, we explicitly defined an output template within the prompt (TEMPLATE) and requested adherence to Python's syntax for multi-line comments (Docstrings).
Listing <ref> only contains example 1 of the few-shot prompt. 
Examples 2 and 3 are available in our replication package <cit.>.
We acknowledge that the prompt can be further optimized and generalized to multiple programming languages and application domains.
We considered its performance and complexity sufficient for our research goal.



  §.§.§ Tool of the Ad-hoc Prompt Group





We focused on ide extensions as developers usually use ide for generating code documentation.
We chose vscode for its extensive customization capabilities through various extensions maintained by an open-source community. 
During our study, the extension marketplace already offered multiple ChatGPT-like tools that we examined.
Additionally, vscode supports multiple programming languages, including Python.

We selected the ad-hoc prompt tool based on two criteria.
First, it needed to provide a user interface similar to the ChatGPT interface, allowing users to engage in a conversation with the software. 
This approach involves at least one self-written message to the model and copying and pasting the generated comment from the chat interface to the code.
This criterion was crucial as we aimed to compare the user experience for this classic Q&A-based interaction with the one for the predefined prompt tool, which generates and inserts code documentation with a button click.
Second, the tool had to support and behave like the original GPT-4 api to handle any confounding factors related to the use of different llm models.

After a thorough testing of available extensions, we opted for ChatGPT by zhang-renyang <cit.> in version 1.6.62. 
This extension met all specified requirements.
It offers a small prompt field (Figure <ref>) and a larger ChatGPT-like chat window within the ide.
Our test showed that it mimics GPT-4.
It was one of the most popular GPT extensions in the marketplace during our study period.



  §.§.§ Tool of the Predefined Prompt Group




Our design objective for the predefined prompt group was to minimize the steps required for developers to generate a high-quality Docstring comment for a specific code section. 
We determined that marking the code within the ide's code editor and selecting the Generate a comment action from the context menu should suffice, requiring three clicks to generate documentation (Figure <ref>).
Alternatively, the developer can choose this action through vscode's command field. 
Hence, inexperienced and experienced vscode users can use the primary feature of the tool with their preferred workflow.
A secondary feature allows developers to customize the predefined prompt via the Edit default prompt action, enabling them to tailor the tool output to their preferred format.


We followed the vscode guidelines to create the TypeScript-based extension <cit.> and utilized the axios package to implement calls to the OpenAI api.
Based on the input prompt, the api returns a code comment generated by GPT-4.
The extension automatically inserts this documentation above the previously selected code.
To set up the extension, developers must enter their OpenAI api token into a configuration field.
This token is stored locally on the developer's machine.
For the experiment, we pre-configured the tool with our token.


We used GPT-4 version gpt-4-0613 for both tools. 
Generating code documentation with this model took, on average, four seconds.
The vscode version was 1.85.
The predefined prompt tool is included in our replication package <cit.> and also available in the vscode marketplace <cit.>.



 §.§ Experiment Execution and Participants




The experiment took place in October and November 2023, involving 50 participants: 25 per ad-hoc and predefined prompt group. 
20 participants were professionals, including 12 software engineers, four data scientists, and four with other software development roles. 17 worked in the domains of scientific computing and three in finance.
These participants took the expert perspective due to their professional experience with Python programming and, eventually, the application domain.
The remaining 30 participants were students in computer science or related subjects, taking the newcomer perspective.
We recruited participants through personal contacts in academia and industry to ensure that the participants had at least some experience with Python and vscode. 
We conducted 33 experiments online and 17 offline at the euxfel campus in Schenefeld, Germany, with the first author leading the experiment and the second author assisting.

On a scale from 1 (very low) to 5 (very high), the mean value and standard deviation of the Python experience among professionals were 3.7 (1.06) in the ad-hoc and 4.5 (0.71) in the predefined prompt group.
The values among students were notably lower, with 2.53 (0.83) in the ad-hoc and 2.47 (1.19) in the predefined prompt group.
This difference confirms the expert/newcomer setting.

All participants were familiar with vscode, which was important to mitigate biases due to the tool setup.
Mean values and standard deviations in the ad-hoc/predefined prompt group among professionals were 2.9 (0.99) and 3.5 (1.18); among students they were 3.07 (1.03) and 3.07 (1.28).

At the beginning of each session, we trained each participant in using the respective tool to mitigate learning bias.
We introduced them to the tool features, generated documentation for an example function, and answered tool-related questions.
Afterward, we started the experiment.
Due to manually writing prompts, ad-hoc prompt group participants completed the experiment after 23:53 minutes on average, while predefined prompt group participants required only 19:58 minutes.



 §.§ Data Analysis




Our experiment comprised quantitative and qualitative data analyses.
We report totals, mean values, and standard deviations where applicable.
Furthermore, we employed inferential statistics to test for significant differences between the groups' ratings of the six documentation quality dimensions. 
Depending on the statistical assumptions fulfilled by the respective variables, we utilized the parametric Welch's t test or the non-parametric mwu test <cit.>.

The assumptions for both tests were that the collected data points were quantitative and independent.
The first assumption was fulfilled as we provided a five-point answer scale for each question related to a quality dimension.
The second assumption was fulfilled as we conducted the experiment with each participant individually, and we asked participants to not share information with others to avoid external influences.
The third assumption of the Welch's t test is a normal distribution of data points, which we tested with the Shapiro-Wilk test <cit.>.
If a normal distribution was not met for a specific quality dimension, we applied the mwu test instead.
Besides reporting the p-values for statistical significance, we also report the effect sizes to indicate practical significance.
We particularly applied Cliff's δ <cit.>, which describes the overlap of two groups of ordinal numbers.
Values between 0.0 and 0.8 indicate a low to medium effect size, whereas values from 0.8 represent a large effect size, meaning a large difference between both groups.


To quantify the experience of developers using the experiment tools, we utilized the 26 seven-point scales of the ueq <cit.> and its official data analysis tool <cit.>.
This tool provides descriptive (mean values and standard deviations) and inferential statistics (t test) to compare user experience ratings for the two studied ide extensions.

Each of the 26 scales is related to one of six quality categories.
The first category, Attractiveness, represents the pure valence of and overall satisfaction with a tool.
Efficiency, Perspicuity, and Dependability are the pragmatic, goal-directed categories, and Stimulation and Novelty are hedonistic, not goal-directed categories.
We chose Efficiency, Perspicuity, Dependability, and Novelty to measure tool-related experience, as these categories focus on tool interactions.

To evaluate participants' assessment of the quality of generated documentation, we used the Stimulation category as it focuses on the output of a tool.
We enhanced this analysis with qualitative insights made by manually analyzing participants' free-text answers to the questions What did you like most about the tool? and  What would you change about the tool?, placed at the end of the questionnaire. 
Two authors independently conducted Open Coding for all answers and discussed their codes to achieve consensus <cit.>.
Furthermore, we report relevant remarks expressed by participants during the experiment.










§ PROMPTING FOR CODE DOCUMENTATION (RQ1)





 §.§ Patterns in Ad-hoc Prompts




The prompts entered by the ad-hoc prompt group varied in detail and keywords. 
About half of the participants had to re-enter a different prompt after their first attempt to generate a code comment.
Among the failed attempts were prompts such as Explain the function and Describe what this function does, which summarized the code in longer prose text.


We categorized the 50 prompts that led to a successful generation of code comments for both code functions into five prompt patterns, listed in Table <ref>.
These patterns commonly included documentation-specific keywords, i.e., comment (34x), Docstring (12), and Python-conform description (1).
Also, slightly longer prompts that included the comprehension-focused terms explain (2) and summarize (1) led to appropriate function comments.
Professionals used the Docstring term more often (9x) than students (3x), which is in line with the higher Python experience of most participating professionals.


Successful prompts were generally short, with a mean length and standard deviation of 7.1 (2.09) words among professionals and 6.7 (2.62) words among students.
The shortest prompt was Generate docstring.
None of the participants enhanced their prompts with examples or templates.



 §.§ Perceived Quality of Generated Documentation




Participants rated the documentation generated by the ad-hoc and predefined prompt tools based on six five-point-scale questions.
All participants were aware of the expected format of a Python Docstring, as they were familiar with Python. 
Moreover, before using the tool, they manually created a function comment based on a Docstring template.
Table <ref> displays the ratings per group, subgroup (student vs. professional), code function, and quality dimension, along with the p-values <cit.> and effect sizes <cit.>.


We performed several trials to determine the appropriate statistical test (see section <ref> for details). 
We rejected the null hypothesis of the Shapiro-Wilk test for all groups and dimensions, indicating a non-normal data distribution, likely due to small sample sizes. 
We chose the mwu test for all comparisons due to its suitability for non-normally distributed data <cit.>.
The null hypothesis of this test indicates no statistically significant difference between the mean values of both groups for a specific dimension. 
We rejected this null hypothesis if the p-value was lower than 0.05.
We noted that in all cases when the null hypothesis was rejected, the effect size δ was large (0.8 or higher), indicating practical significance <cit.>.




Comparing all participants in the ad-hoc and predefined prompt groups, we observed several statistically significant differences.
For function 1, we found significant differences in the dimensions of Readability (4.12/4.72) and Unnecessary Information (3.28/4.32).
The predefined prompt tool consistently provided concise Docstrings that participants perceived as more readable and contained less non-informative content compared to the output of the ad-hoc prompt tool.
For function 2, significant differences were found in four dimensions, with the predefined prompt tool consistently receiving higher ratings than the ad-hoc prompt tool.
The differences in Readability (3.52/4.68) and Unnecessary Information (2.84/4.68) reinforced the conclusions drawn from function 1.

We attribute the ratings of Usefulness (3.52/4.6) to the often excessive output length of the ad-hoc prompt tool.
Participants noted that longer, prose documentation could impede development workflows, as these comments require more time to read while providing little additional information.
Helpfulness was rated high in both groups (4.0/4.6), with some participants suggesting that the explanations provided by both tools were either too complicated or lacked the required level of detail.
Across both groups, participants rated Grammatical Correctness and Missing Information very high.


When we compared documentation ratings between students and professionals, we observed similar statistically significant differences in the ad-hoc and predefined prompt groups for students, while professionals did not exhibit significant differences.
Among students, we observed significant differences for function 1 in the dimensions Readability (4.0/4.66), Unnecessary Information (3.33/4.86), and Usefulness (3.93/4.6).
The different expectations regarding the documentation content contributed to the disparity in Usefulness, with some students preferring the longer summarizations by the ad-hoc prompt tool, while most others expected a structured comment as taught in university.

For function 2, students in the predefined prompt group consistently provided significantly better ratings for Readability (3.2/4.6), Unnecessary Information (2.66/4.86), Usefulness (3.13/4.73), and Helpfulness (3.66/4.6) compared to students in the ad-hoc prompt group.
Unnecessary Information had the highest differences, likely due to the unstructured and lengthy comments generated via the ad-hoc prompts.

Among the professionals, we found no statistically significant differences between the ad-hoc and predefined prompt groups.
However, three dimensions showed large effect sizes (≥ 0.8), indicating a practical significance.
These differences concerned Readability for functions 1 (4.3/4.8) and 2 (4.0/4.8), and Unnecessary Information for function 2 (3.1/4.4).
This indicates that professionals also found the comments of the ad-hoc prompt tool to contain non-informative content, impairing readability.
Conversely, the ad-hoc prompt group provided a more positive rating for the dimension Missing Information (4.7/4.0) for function 1.
Professionals expected more detailed descriptions, which the concise comments by the predefined prompt tool did not provide.
Furthermore, as professionals used the Docstring keyword more often than students in their prompts (Table <ref>), the ad-hoc prompts in these cases resulted in similar documentation to the predefined prompt tool.



 §.§ Code Comprehension




We analyzed how the llm-generated documentation influenced the comprehension of the code.
Initially, participants rated their perceived comprehension of the respective code function without available documentation, on a scale from 1 (not at all) to 5 (very well).
For function 1, the mean and standard deviation for these ratings were 3.6 (0.96) by the ad-hoc and 3.84 (1.11) by the predefined prompt group, indicating a similar good understanding.
The assessment for function 2 was lower, with 2.4 (0.96) by the ad-hoc and 2.92 (0.81) by the predefined prompt group.
Hence, the predefined prompt group expressed slightly higher confidence in the comprehension than the ad-hoc prompt group, despite similar stated Python experience.
Across both groups and functions, professionals stated a higher comprehension than students, which aligns with their higher Python experience.


We assessed participants' actual comprehension through True/False questions about the code (Table <ref>).
We counted the number of participants who correctly answered these questions before and after generating the documentation. 
Participants could also choose the option Not enough information available to answer the question, which we treated as an incorrect answer.

Before using the tools, most participants in both groups answered the two questions for function 1 correctly (Q1: 19+23; Q2: 23+24), as expected for this short and easy-categorized function.
After using the tool, the correct answers for question 1 increased (Q1: 22+24), especially among students, while the ones for question 2 slightly decreased (Q2: 21+20) for both students and professionals.
Thinking-aloud observations revealed that many participants were confused by the less technical documentation generated by both tools regarding this question, as it did not include the term iterable.

The number of correct answers to function 2 before using the tool was lower (Q1: 10+14; Q2: 15+13), reflecting its higher complexity. 
In both groups, correct answers increased after using the tools (Q1: 22+18; Q2: 21+21). 
Students had the highest increases: in the ad-hoc prompt group for Q1 from 5 to 13, and in the predefined prompt group for Q2 from 5 to 14. 
The increase in comprehension among professionals was marginal, except for the correct Q1 answers in the ad-hoc prompt group, which increased from 5 to 9.
This data shows that the availability of documentation helped inexperienced developers to better comprehend the code, while the effect on professionals was mixed.


Answering RQ1, we conclude that developers were able to generate code documentation using ad-hoc prompts without prompt engineering techniques, often by providing relevant documentation keywords.
However, particularly less experienced students initially did not use such keywords and generated lengthy, less readable, and less useful code explanations instead.

This is reflected in participants rating the quality of generated documentation.
We observed statistically significant differences exclusively among students, who rated the readability, conciseness, usefulness, and helpfulness of comments generated via the predefined few-shot prompt higher than those created via ad-hoc prompts.
Although we observed large effect sizes for some quality dimensions among professionals, this subgroup was generally less critical as they often generated documentation of the required quality and format by using specific keywords.

We also conclude that documentation generated by both tools helped (particularly students) comprehend more complex code.
The generated documentation could also lead to confusion, even for professionals.









§ DEVELOPER EXPERIENCES WITH PROMPTING (RQ2)





 §.§ Overall Experience



Participants rated both tools positively in all six categories of the standardized ueq <cit.>, with values ranging from -3 to +3 (Table <ref>).
However, students found the predefined prompt tool more positive in all user experience categories than the ad-hoc prompt tool, with statistically significant differences and large effect sizes in five categories.
Professionals also rated the predefined prompt tool more positively in five categories, without significant differences but a large effect size in one category.

For the pure valence category Attractiveness (1.01 ad-hoc / 1.99 predefined prompt group), the predefined prompt tool received higher ratings than the ad-hoc prompt tool, with a high difference among students (0.64/1.94). 
This indicates an overall better developer experience with the predefined prompt tool.



 §.§ Tool-Related Experience



The most significant difference between the groups was in the category Efficiency (0.7/1.98), with the predefined prompt tool receiving higher ratings, especially among students (0.42/2.12).
Participants praised the ad-hoc prompt tool for its practicality of not having to open ChatGPT in the browser (two students, one professional), flexibility in selecting relevant code (one professional), and custom prompts (one professional).
They noted issues with copy-pasting the tool output to the code panel (six students, four professionals), multiple tries sometimes required for sufficient results (one student), the need to phrase a prompt (one professional), missing keyboard shortcuts (one professional), and the slow output generation, which is primarily caused by the performance of GPT-4 (one student).
Participants who interacted with the predefined prompt tool also noted the slow response (one student, two professionals), but expressed no further positive or negative comments.


The ratings for Perspicuity (1.25/1.87) were higher for the predefined prompt tool, with large effect sizes for students and professionals.
While some participants of the ad-hoc prompt group participants found their tool easy to use (four students, one professional), others reported that they would find it easier to enter the prompt in a separate ide panel instead of the pop-up at the top of the screen (one professional) and that the right-click context menu, which listed the tool features besides non-related ide features, was obscure to them (one professional).
Multiple predefined prompt group participants also found their tool easy to use (eight students, four professionals), highlighting the ease of generating comments without the need for further prompt input besides the code and a short command (one professional).
Nevertheless, they assessed that the selection of relevant code could be facilitated (one professional).
Overall, this assessment aligns with our observation that participants quickly understood both ide extensions but tended to prefer the simplicity of the predefined prompt tool without having to create an effective prompt.


Ratings for Dependability (1.05/1.84) showed strongly diverging views among students (0.85/1.95).
Participants praised the ad-hoc prompt tool for supporting the comment creation workflow (six students, two professionals) and code comprehension (three students, four professionals).
They criticized the lack of prompt templates (three students), that the tool did not always generate code comments (two students, one professional), and the missing opportunity for iterative improvements caused by the limitations of the OpenAI api at the time of our study (one student, one professional).
The predefined prompt tool was also noted for facilitating the workflow (three students, one professional) and code comprehension (four students), as well as for its reliability (two professionals).
However, participants suggested adding a progress bar (one professional), previewing generated comments within the function (one professional), enabling the configuration of the required comment content (one student), adding a disclaimer that the output might be incorrect (one student), implementing a linter that alerts when generated comments became outdated (one professional), and using a locally deployed api for data security (one professional).
Besides these improvement comments, the predefined prompt group was overall satisfied regarding the Dependability aspects, whereas the ad-hoc prompt group, and especially the students, required more enhanced tool support.


Both tools achieved similar positive results concerning Novelty (1.1/1.18), without significant differences.
Interestingly, the ad-hoc prompt tool received a higher rating from professionals (1.38/1.18) for its chat-like interaction within an IDE, which they considered more novel than clicking a button for documentation generation.
Participants expressed no further comments regarding this ueq category.



 §.§ Documentation-Related Experience



For Stimulation (0.82/1.62), predefined prompt tool ratings were higher, especially among students (0.6/1.6).
Concerning the output of the ad-hoc prompt tool, participants noted the detailed explanations (two students, two professionals), the mitigation of personal biases (one student), inspirational value (one professional), and overall high quality (three professionals).
Other participants criticized the output for containing unnecessary information (five students, three professionals), lack of detail regarding complex code (three students), and inconsistent format due to using prompts of varying qualities (two students, one professional).
The predefined prompt tool output received positive ratings for its conciseness (four students, one professional), overall high quality (three professionals), and similar structure across all generated comments (one student, one professional).
Improvements were deemed necessary for the parameter descriptions (one student, one professional), line breaks (one professional), consistent punctuation (one student), and lack of detail for complex code (one student, one professional).
These insights show that students were especially unsatisfied with the ad-hoc prompt tool output, while professionals were overall satisfied with both tools.


Answering RQ2, we conclude that the participants overall preferred a tool that automates code documentation generation with a few clicks while offering options to configure this process.
On average, professionals and students preferred executing predefined prompts over creating ad-hoc prompts.
The simplicity and efficiency of a single button click to receive consistently high-quality documentation were key factors for this assessment.
While the predefined prompt group praised their tool, some missed the flexibility to adjust the documentation depth.
Users of the ad-hoc prompt tool appreciated this flexibility, which often unintentionally resulted in longer and more explanatory comments that may assist code comprehension.








§ DISCUSSION



We summarize the results of our experiment and discuss potential implications for research and practice.



 §.§ Developers are not Necessarily Prompting Experts


Our study indicates that developers are generally neither skilled (RQ1) nor inclined (RQ2) to effectively prompt an llm in a way that it generates concise, readable code documentation. 
This partly aligns with previous studies showing that optimized, predefined prompts outperform ad-hoc prompts <cit.>. 
But it also partly contradicts the underlying assumption that effective optimized prompts, such as few-shot prompts, are trivial and can intuitively be created by developers in their daily work. 
We observed in our experiment that participants interacted with the provided tools intuitively via natural language queries.
However, they were often disappointed by the results as the tools provided outputs that were aligned with their unspecific prompts.
In fact, we even observed that the generated documentation can partly be misleading during comprehending code. 

The lack of prompting skills among developers is unsurprising, as this is not a standard skill taught in universities or commonly practiced in software development.
Therefore, researchers should explore ways to learn effective prompting and to intuitively support conversations with llm, particularly in domains like software engineering where the llm outcome has an impact on other users.
One approach is to compile a catalog of evaluated prompt templates for different well-defined tasks (e.g., documentation of Python functions).
Recently, Torricelli et al. also found such templates beneficial for llm users <cit.>.
Tool vendors could integrate the templates into their tools.
Each template could include instructions for the llm to ask clarifying and contextual questions, e.g., related to the expected documentation style <cit.>.
Additionally, educators could use these prompt templates, together with underlying ideas and hints, to teach prompting skills in schools and universities.
Such templates should only provide guidance and should not hinder human creativity in asking out-of-the-box and follow-up questions to an llm-powered tool.



 §.§ Generating Documentation With LLMs is an Iterative Task


Our results show that the participants rated the quality of documentation generated via the predefined prompt as high.
However, many were not completely satisfied with the documentation quality after this initial iteration.
The first iteration helped to comprehend the code, formulate the initial comment version, and gather ideas to alter and extend the documentation during further iterations.
This indicates that generating code documentation is rather an iterative task, not to be fully automated without the feedback of developers.

To better understand and assist this task, additional research should be conducted to identify and satisfy the preferences of code documentation providers and consumers <cit.>. 
Personal aspects include the depth of explanations, tone of the text, and lyrical style. 
Project aspects include documentation style used <cit.> and knowledge needs of code users.
This research can help vendors of ai-powered code documentation tools align the generated comments with the requirements of their users.
Thus, vendors can focus on applying and optimizing predefined prompts, which participants in our experiment preferred, while also offering additional iterations to customize the output based on preferences and needs.
Furthermore, generating multiple documentation versions at once might also stimulate developer creativity in exploring possibilities to document code <cit.>.



 §.§ Evaluating Code Documentation Quality Requires Human-Centered Quality Metrics


As Hu et al. <cit.> pointed out, current metrics used to assess the quality of ai-generated content do not align with human evaluation regarding code documentation generation.
For instance, the usefulness of generated content <cit.> should be crucial in practice but is barely evaluated in common llm benchmarks. 
Our results suggest that the assessment of code documentation quality depends on multiple factors <cit.>, such as the preferences of documentation consumers, the ratio between documentation conciseness and code complexity, and the availability of certain information and style that developers expect to be included in the documentation.
Current metrics for automatically assessing the quality of generated texts do not incorporate these aspects.
Hence, while automatically generating code documentation is fairly straightforward with llm, an expert assessment is still required to ensure good quality, e.g., during a code review.
It remains unclear how far this assessment can be fully automated, e.g., by other agents or other llm.
Future research should focus on creating and testing quality metrics for generated documentation through documentation analyses, expert surveys, and developer and benchmark studies.



 §.§ Better Code Comprehension Requires Support by Personalized LLM-Generated Explanations On The Fly


The results of our code comprehension tasks show that llm can support developers in understanding code.
Especially, the longer explanations provided by the ad-hoc prompt tool in response to vague prompts helped participants understand the given code function.
However, such explanations can also mislead developers if the generated output does not consider the information needs <cit.> or provide the particular knowledge nuggets needed for the comprehension task <cit.>.

Therefore, similar to the work of Nam et al. <cit.>, we propose that vendors of ide-integrated code comprehension tools should offer multiple code-related aspects that tool users can explore during the comprehension process.
Examples include the concepts mentioned in the code or the typical code usage <cit.>, but could also extend to explaining the code in a personalized way.
Researchers should survey developers to identify additional aspects and test their relevance in practice.



 §.§ Practitioners May Have Specific LLM Requirements


Participants in our study expressed multiple requirements for an llm-powered documentation generation tool.
For example, participants expressed concerns regarding security risks associated with using closed-source, externally hosted models and highlighted the extended time required for generating comments via the llm model we used.

To address these issues, tool vendors should support a variety of llm within a single tool, including smaller, open-source models, to better meet the needs of organizations operating with sensitive data and under specific constraints.
One of the research challenges is that different models may respond variably to the same prompt, impacting the consistency and reliability of generated outputs.






§ THREATS TO VALIDITY





 §.§ Construct Validity





(T01) Our study exclusively employed human evaluation techniques to assess code documentation quality, introducing a mono-method bias <cit.>.
Although automated metrics are used in other studies, we opted for human evaluation due to its limited correlation with automated metrics <cit.>, and considered it beneficial for a more comprehensive understanding.



 §.§ Internal Validity



(T02) We unintentionally excluded the Understandable scale from our ueq, which may compromise the Perspicuity category's accuracy. 
Remedying this, we calculated Perspicuity scores with and without the scale's median value, finding only marginal differences. 
Despite this omission, participant feedback indicated an adequate understanding of the tool.
(T03) Customizability of the predefined prompt could have positively influenced participant satisfaction and tool assessment. 
However, allowing customization could have affected the comparability of assessments among the predefined prompt group. 
(T04) Lack of significant differences in professional ratings may be attributed to the small sample size of ten professionals per group. 
Nevertheless, descriptive statistics reveal fewer differences among the professionals' ratings compared to students, likely due to their more targeted use of keywords in ad-hoc prompts.
(T05) A potential language barrier existed given the non-native English-speaking background of most participants. 
However, this was deemed a minor threat as all were able to produce coherent prompts in English.



 §.§ External Validity




(T06) Our controlled lab setup, where developers documented unfamiliar code, may not accurately represent typical documentation practices. 
However, this setting was intended to eliminate variability and enable statistical analysis. 
(T07) The professionals in our study predominantly came from euxfel, with specific expertise in Python, which might influence the results due to familiarity with the experimental code. 
This was seen as potentially beneficial, providing insight into how domain knowledge of one experiment group might improve their prompt creation.
(T08) The findings' generalizability is limited, yet they corroborate and broaden the insights from similar studies that involved different methodologies.



 §.§ Conclusion Validity




(T09) The selection of utility functions as code candidates might limit the applicability of our results to more complex coding tasks.
We chose these functions based on a pilot study confirming the difficulty of comprehending challenging code under time constraints.
(T10) Variations in llm outputs due to inherent randomness <cit.> may have influenced assessments in the predefined prompt group, though differences in outputs were marginal.
(T11) Dividing the experimental and control groups into professionals (experts) and students (newcomers) reduced overall sample sizes, weakening statistical power but allowing for a more detailed analysis of differing perceptions between these groups.
The separation between experts and newcomers was confirmed by the participants' self-perceived Python experience.
Hence, we believe our findings can be generalized to a broader population.
Nevertheless, developers with different expertise in conversational agents might behave differently.







§ RELATED WORK





 §.§ Prompting Skills of Practitioners



Prompt engineering has been pivotal since the introduction of few-shot prompting <cit.>. 
Ahmed et al. demonstrated its effectiveness in code summarization <cit.>.
They highlighted that applying a few-shot prompt can outperform traditional models like CodeBERT <cit.>, while zero- and one-shot prompts were less effective, aligning with findings of Geng et al. <cit.>. 
Meanwhile, subsequent techniques, like chain-of-thought prompting <cit.> and active prompts <cit.>, have emerged.

Prompt engineering remains a skill requiring practice <cit.>.
llm providers and researchers have published guidelines and prompt pattern catalogs for effective llm interaction across various domains, including healthcare <cit.>.
Our study is among the first to examine if and how well developers utilize prompt-related knowledge in software engineering tasks.



 §.§ LLM-Powered Developer Assistants



Recent studies on ai programming assistants show that tools like GitHub Copilot <cit.> boost productivity, reduce keystrokes, and assist in syntax recall <cit.>.
However, users desire tool enhancements for direct feedback, personalized output, and improved code context understanding <cit.>.
Currently, they often avoid these tools due to unmet requirements and lack of control.
For example, Copilot often fails to grasp instructions for code adjustments unless precisely specified <cit.>.
However, tool shortcuts to optimize an initial prompt can also nudge users to explore fewer prompts and lead to less diverse output <cit.>.
We can confirm these findings with our study.
While our participants overall preferred the interaction with and the output of the predefined prompt tool, the more exploratory ad-hoc prompts often led to more extensive output that supported code comprehension.



 §.§ Code Documentation Generation in the IDE




In the last decade, researchers have developed various methods for generating code documentation <cit.>, including template-based <cit.>, information retrieval <cit.>, and, more recently, deep learning techniques <cit.>.
Especially deep learning approaches trained on large datasets have shown high performance in code documentation generation and enabled better scalability and more complete results than the other techniques <cit.>.

We found many available ide extensions that use these ai models to facilitate developer workflows.
However, they were conceptualized as general-purpose tools and lacked code documentation features.
In the academic realm, the vscode extension Divinator was the most relevant tool to our project <cit.>. 
It provides short code summaries in multiple programming languages but has limited performance and interactivity.
With our study, we aimed to counter this lack of ide features related to code documentation generation.



 §.§ Evaluating Documentation Quality



Empirical research on software documentation quality is an active field that focuses on various artifacts, like api reference documentation <cit.> or README files <cit.>, and the perspectives of documentation writers <cit.>.
Studies on the evaluation of ai-generated documentation usually focus on automated metrics like BLEU, ROUGE, and METEOR <cit.>.
Hu et al. compared such automated metrics with human evaluations on six documentation quality dimensions <cit.> and found that automated metrics often misalign with human judgment <cit.>. 
Consequently, our study benefited from their human-centered metric by capturing nuanced aspects in the generated code documentation that automated approaches may have missed.







§ CONCLUSION




Recently, many llm have emerged, offering the potential to automate developer tasks, including creating code documentation.
However, it remained unclear how effectively developers could prompt llm to generate useful documentation.

We studied the interactions of developers with and their perception of llm-powered ide extensions during a controlled experiment with professionals and computer science students.
To generate documentation for two Python functions, the experimental group freely prompted an llm and the control group applied a predefined few-shot prompt.

Our results revealed that students, who had relatively low Python experience, preferred the guidance of the few-shot prompt over ad-hoc prompting.
They rated the documentation generated by predefined prompts significantly higher in quality, particularly regarding readability, conciseness, usefulness, and helpfulness.
Professionals were more adept than students at including Python-specific keywords in their ad-hoc prompts, resulting in the generation of higher-quality documentation.
Consequently, they enjoyed the flexibility of ad-hoc prompting more than students, even if they did not apply prompt engineering techniques.
Overall, both types of llm interactions improved the code comprehension of the study participants, but the participants often viewed the generated documentation as an intermediate result that needed iterative improvement.

We hope our findings encourage researchers to replicate this study <cit.> in diverse settings, aiming to improve developers' prompting skills and ai-powered tool support.




§ ACKNOWLEDGEMENT

We thank all participants of our experiment.
We acknowledge the support by DASHH (Data Science in Hamburg - Helmholtz Graduate School for the Structure of Matter) with the Grant-No. HIDSS-0002.

IEEEtran




 Paper Title: A Dataset for Multi-intensity Continuous Human Activity Recognition through Passive Sensing.



Enhancing Deep Hedging of Options with Implied Volatility Surface Feedback InformationWe thank John Hull and Zissis Poulos for valuable feedback about this work. Gauthier is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC, RGPIN-2019-04029), a professorship funded by HEC Montréal, and the HEC Montréal Foundation. Godin is funded by NSERC (RGPIN-2024-04593).
 
    [
     
August 5, 2024
=============================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Human activity recognition (HAR) is essential in healthcare, elder care, security, and human-computer interaction. The use of precise sensor data to identify activities passively and continuously makes HAR accessible and ubiquitous. Specifically, millimeter wave (mmWave) radar is promising for passive and continuous HAR due to its ability to penetrate non-metallic materials and provide high-resolution wireless sensing. Although mmWave sensors are effective at capturing macro-scale activities, like exercising, they fail to capture micro-scale activities, such as typing. In this paper, we introduce [<https://github.com/arghasen10/mmdoppler> (Access: August 5, 2024)], a novel dataset that utilizes off-the-shelf (COTS) mmWave radar in order to capture both macro and micro-scale human movements using a machine-learning driven signal processing pipeline. The dataset includes seven subjects performing 19 distinct activities and employs adaptive doppler resolution to enhance activity recognition. By adjusting the radar's doppler resolution based on the activity type, our system captures subtle movements more precisely. includes range-doppler heatmaps, offering detailed motion dynamics, with data collected in a controlled environment with single as well as multiple subjects performing activities simultaneously. The dataset aims to bridge the gap in HAR systems by providing a more comprehensive and detailed resource for improving the robustness and accuracy of mmWave radar activity recognition.

This paper introduces , a novel dataset of human activities collected using a commercial off-the-shelf (COTS) Millimeter Wave (mmWave) radar, utilizing adaptive Doppler resolution to capture detailed and precise motion data. This dataset encompasses various activities, including macro-scale movements like walking and squats and micro-scale actions like typing and sitting. The advanced mmWave radar technology offers unparalleled motion resolution while safeguarding privacy, as it does not rely on optical imaging. Our dataset provides a comprehensive foundation for developing sophisticated machine learning and deep learning models for activity recognition. Indoor human activities were recorded for 5 subjects performing 19 distinct activities across 3 unique environments. Preliminary analyses reveal significant improvements in the accuracy and reliability of activity detection approaches, underscoring the transformative potential of adaptive Doppler resolution in contactless sensing applications. By making this dataset publicly accessible, we aim to drive innovation and accelerate progress in real-time, non-invasive activity recognition systems, thereby establishing a new benchmark for future research in human-computer interaction, elevation of standard of living, and health monitoring. This work paves the way for the next generation of privacy-preserving, high-precision activity recognition technologies.










§ INTRODUCTION


Think of the possibilities when technology seamlessly and passively tracks your movements, capturing everything from the subtle flick of a finger to a grand gesture of a hand wave (macro-scale). In recent years, millimeter wave (mmWave) radar technology has made this vision a reality. A mmWave radar's ability to penetrate non-metallic materials, resilience to environmental changes, and high-resolution sensing capabilities distinguish it from other radar technologies. Traditional methods of human activity recognition (HAR) rely on sensors like LiDAR or depth cameras, which excel at capturing large movements such as walking or running <cit.>. However, these methods often fail to detect more nuanced activities like typing or subtle hand movements. Therefore, advanced sensor techniques are necessary for accurately capturing both macro- and micro-scale human movements.


To address this challenge, we introduce , an innovative dataset of human activities collected using a Commercial off-the-shelf (COTS) mmWave radar. This dataset bridges the gap between macro and micro-scale activity recognition by capturing activities through their range-doppler signatures, along with point cloud data. A point cloud represents a collection of three-dimensional points reflecting radar signals from objects in the environment, providing detailed spatial information. The range-doppler heatmaps are 2D image representations where the abscissa represents the range (location of human subjects), and the ordinate indicates the doppler speed at which subjects are moving. These heatmaps combine range and doppler information from radar signals, offering a detailed view of motion dynamics over time. By integrating these data types, encompasses a wide range of activities—from large actions like walking and squats to fine-grained behaviors like typing and sitting. Seven subjects performed 19 different activities in 3 unique indoor settings, providing a rich and diverse dataset for advancing HAR research.











Capturing the subtle movements of micro-scale activities even with mmWave radar presents a significant challenge. To overcome this, we utilize an innovative approach with adaptive doppler resolution instead of capturing only the pointcloud dataset as done in the prior works <cit.>. Higher doppler resolution allows for precise detection of small movement changes, while lower doppler resolution is sufficient for detecting larger movements. However, higher doppler resolution also introduces more noise. Therefore, by optimizing doppler resolution based on the activity type, our system effectively captures the nuances of various movements for .

For macro-scale activities characterized by significant movement, such as walking or running, a lower doppler resolution is used to efficiently capture the prominent velocity changes without needing fine detail. Conversely, for micro-scale activities involving minimal movement, like typing or subtle hand gestures, a higher doppler resolution is employed, allowing the radar to detect subtle velocity changes with greater detail and accuracy. This adaptive doppler resolution approach facilitates the robustness and precision of HAR systems, making them more effective across a wide range of applications that require both macro and micro-scale movement recognition.









That said, can be characterized by the following key features:


    ∙ Range-doppler information: is specifically curated with range-doppler heatmaps of different activities of daily leaving. This is in contrast to the traditional activity signature datasets with only point clouds generated by mmWave radar. 

    ∙ Adaptive Doppler resolution: as we observe that the macro and micro-scale activities can be better captured by dynamically switching the radar sensitivity. , therefore, is curated with adaptive doppler resolution.

    ∙ Diverse Activity Set: We have included a wide array of 19 different activity classes from Activities of Daily Living (ADLs), Instrumental Activities of Daily Living (IADLs) <cit.>, and daily indoor exercises – (i) macro activities like walking, running, jumping, clapping, lunges, squats, waving, vacuum cleaning, folding clothes, changing clothes, and (ii) micro activities like laptop-typing, phone-talking, phone-typing, sitting, playing guitar, eating food, combing hair, brushing teeth, and drinking water to ensure comprehensive coverage of daily human behaviors. 

    ∙ Real-world setups: The data is collected mimicking diverse scenarios of real indoor environments with the presence of furniture and other objects usually found in an indoor setup. This promotes the real-world applicability of any system evaluated on this dataset. 

    ∙Controlled Environment: To avoid unaccountable noise signatures, Data collection was conducted in a controlled indoor environment. This also ensures the quality and consistency of the data. comprises an effective duration of 23100 seconds of data, 27% more than the closest largest dataset in existence <cit.>.

    ∙ Multiple Subjects: uniquely includes multiple subjects performing various activities simultaneously, unlike traditional datasets that typically focus on single-subject activities <cit.>. This feature allows our dataset to account for the complexity and variability of real-world scenarios where interactions between multiple individuals are common.



As a result, our dataset provides a valuable resource for researchers and developers working on HAR systems for continuous passive sensing. Combining adaptive doppler resolution and range-doppler heatmaps enables more accurate and robust activity recognition across a wide range of activities. By combining mmWave radar technology with point cloud data, this dataset overcomes the limitations of previous HAR systems solely based on point clouds. Making this dataset public will enable further research and development of innovative solutions that leverage the capabilities of mmWave radar to comprehensively and reliably detect continuous human activity through passive sensing.


§ RELATED WORK

Human Activity (HA) detection has been extensively studied over the past few decades. However, the methods and technologies used for assessment have evolved significantly in response to technological advancements. Camera-based <cit.> approaches have been leveraged for quite some time. However, despite being effective, they primarily suffer from privacy concerns or poor lighting conditions. Studies have used wearables for active sensing to detect human activity <cit.>. However, these methods are not necessarily seamless or pervasive. Alternative passive sensing methods, such as those based on acoustics and radio frequency, offer promising solutions. Passive acoustic sensing <cit.> uses audio chirps detected by microphones to study activities but is susceptible to environmental noise and interference <cit.>. Radio-frequency (RF) sensing, using WiFi <cit.>, RFIDs <cit.>, and UWB radars <cit.>, captures human dynamics by analyzing changes in radio waves <cit.>. WiFi Channel State Information (CSI) has been widely explored <cit.> but is complex and affected by environmental factors <cit.>. FMCW techniques with specialized hardware offer higher resolution but are costly <cit.>. COTS mmWave sensors, like IWR1642 <cit.>, provide better range resolution and can detect fine movements <cit.>. Studies using mmWave sensors rely on emitted chirps to capture activity signatures <cit.>, with features like point clouds and range-doppler proving effective for movement assessment <cit.>. While WiFi and UWB-based methods face various challenges, mmWave sensors offer a robust and potentially fine-grained solution for activity assessment.

Over the past few years, numerous datasets have been developed to support human activity detection research, utilizing various sensing modalities. Traditional datasets often rely on wearable sensors <cit.> and video data <cit.> to capture activities. Dataset using a depth camera <cit.> instead of an RGB camera, can also be seen in the existing works. Similarly, IMU sensors (such as accelerometers, gyroscopes, etc.) using inertial measurement units to detect various physical activities have been used to curate dataset <cit.> for human activity assessment.  Acoustic sensing datasets <cit.>, on the other hand, employ microphones to detect activities through sound. RFID-based datasets <cit.> capture activities by tracking RFID tag movements. WiFi Channel State Information (CSI) <cit.>, sometimes with multimodal information <cit.>, has also been used to capture the impact of human movements on WiFi signals. Despite their utility, these datasets often have limitations such as privacy concerns, poor lighting conditions, acoustic noise, and the need for participants to wear or interact with devices constantly.

More recently, datasets generated by mmWave radar, more specifically, FMCW mmWave radars, have gained attention due to their non-intrusive nature and high resolution. These datasets capture fine-grained details of human movements by analyzing the frequency shifts and distances of reflected radar signals. As mentioned earlier, these datasets can be categorized primarily into two types: point clouds and range-doppler heat maps. Within the first category, multiple point cloud <cit.> datasets exist for studying both macro and micro-level human activities or gestures. Interestingly, these datasets primarily focus on single-subject activities. However, the datasets for human activity assessment that capture range-doppler information are not available to the best of our knowledge. We further compare the available mmWave-based datasets with in terms of relevant features in Table <ref>. Interestingly, recent studies <cit.> have shown that range-doppler heatmaps are significantly effective over point clouds for activity assessment. As a result of using range-doppler information for human activity detection, new avenues are opened for unobtrusive and continuous monitoring, enhancing both the scope and accuracy of the recognition of human activity.










































































































































§ DATASET

 

















































As shown in  <ref>, our data collection setup is built on a Commercial off-the-shelf (COTS) millimeter wave radar, the  <cit.>. The system was tested in three distinct rooms to capture a variety of environments:

    
  * R1: An office cabin of size 4 × 3 m^2
    
  * R2: A classroom of size 8 × 5 m^2
    
  * R3: A laboratory of size 12 × 6.5 m^2

Refer to  <ref>(d), <ref>(e), and <ref>(f) for the specific room setups.

The ground truth for each subject's activity was meticulously annotated with the help of video footage captured using a USB camera. This ensured accurate labelling and high-quality data. The sensor data are transmitted via a USB cable at a baud rate of 921600 to a Dell Inspiron Laptop having Intel(R) Core(TM) i7-8550U CPU with 16 GB of RAM. 






 §.§ Radar Configuration

The IWR1642BOOST radar is configured with two transmitter and four receiver antennas, operating at frequencies ranging from 77 to 81 GHz, providing a bandwidth of 4 GHz. We have employed two distinct radar configurations tailored for different use cases: macro-scale activities and micro-scale activities, as detailed in  <ref>.

For macro-scale activities, the radar configuration is optimized to capture larger movements. We set the frame periodicity to 200 milliseconds, resulting in a frame rate of 5 frames per second (FPS). This configuration enables the transmission of larger range-Doppler heatmaps, with a matrix size of 16 × 256, via USB. The range resolution achieved is 4.36 cm, with a maximum unambiguous range of 9.02 meters. Additionally, this setup can measure a maximum radial velocity of 1 m/s, with a doppler resolution of 0.13 m/s. The sensor transmits 32 chirps per frame, which is suitable for capturing the dynamics of macro-scale activities.

For micro-scale activities, where subtle movements need to be accurately detected, we employ a higher Doppler resolution. The Doppler resolution is set to 0.01 m/s to ensure precise detection of minimal movements. The range-doppler heatmap for this configuration is of size 128 × 64, with 128 number of chirps, supporting a frame rate of 2 FPS. This setup is particularly effective for capturing fine-grained details of micro-scale activities, such as typing or subtle micro-movements.

These configurations, with their respective range and Doppler resolutions, are meticulously designed to optimize the radar's performance for different scales of activity, ensuring accurate and robust data collection for our HAR dataset.







 §.§ Data Collection Setup

Data collection was conducted with 7 subjects (3 female and 4 male), aged between 23 and 35. The total duration of data collection spanned 7 hours, covering 19 different activity classes, encompassing both macro and micro activities. Specifically, we collected 49k macro range-doppler samples and 25k micro range-doppler samples, along with 75k pointcloud data samples.

The majority of the data was collected in a controlled environment where subjects were instructed to perform specific activities. Additionally, to introduce variability and capture real-world scenarios, we included some sessions where subjects could choose and perform any activity from the predefined set in an uncontrolled fashion. This dual approach allowed us to experiment with both controlled and in-the-wild setups, enhancing the dataset's robustness and applicability.

We utilized the mmWave-Demo-Visualizer tool from Texas Instruments <cit.>, with a custom patch implemented to extract raw data, including range-Doppler heatmaps, under different radar configurations. Human activities were meticulously annotated using video footage captured by an additional USB camera installed in the room. Two volunteers assisted in annotating the activities, ensuring high accuracy and reliability of the labels. As a result of this comprehensive data collection methodology, a diverse and well-annotated dataset is provided for development and evaluation of HAR systems using a COTS mmWave radar.



 §.§ License and Consent

The dataset is free to download and can be used with GNU General Public License for non-commercial purposes. The participants received $20 per session as remuneration during the data collection. The experimental procedures were approved by the Ethical Review Committee at IIT Kharagpur, India, with the Approval Number: IIT/SRIC/DEAN/2023, dated July
31, 2023, under the study title: “Human Activity Monitoring in Pervasive Sensing Setup".



§ OBSERVATIONS AND BENCHMARK

We consider 19 different activity classes from Activities of Daily Living (ADLs), Instrumental Activities of Daily Living (IADLs) <cit.>, and daily indoor exercises – (i) macro activities like walking, running, jumping, clapping, lunges, squats, waving, vacuum cleaning, folding clothes, changing clothes, and (ii) micro activities like laptop-typing, phone-talking, phone-typing, sitting, playing guitar, eating food, combing hair, brushing teeth, and drinking water. In contrast to the existing literature that primarily uses voxelized pointcloud <cit.> or 1D doppler <cit.>, in this paper, we explore range-doppler 2D heatmaps for activity classification; the primary motive is to find a parameter that can detect both macro and micro activities simultaneously from different users. 



 





 §.§ Observations from the heatmaps

We perform a set of observations that establish the effectiveness of the range-doppler signatures for capturing diverse activity scenarios. We detail these observations below.


  §.§.§ Feasibility study for range-doppler

 <ref> shows the standard deviation in the range-doppler heatmaps captured during the activity. Notably, standard deviation technique removes static powers (-3 to 2 doppler bins) in the heatmap; thus, we see a low-power value in these doppler bins. We observe that each activity has different signatures captured by the range-doppler heatmap. Although the plotted standard deviation looks similar for some activity pairs with similar body movements (like walking/running, jumping/lunges), there are temporal changes in the heatmaps; for example, “running” induces a faster change than “walking”. Thus, combining the observations from range, doppler, and time, we can get different signatures. Notably, the macro activities have more robust patterns due to the magnitude of movement involved. Even though the micro activities have relatively weaker signatures, they can be distinctively captured with a higher doppler resolution (-64 to +64 doppler bins, in contrast to -8 to +8 doppler bins used for macro activities).




  §.§.§ Impact of radar configurations on determining a subject's activity

To understand how the radar configuration affects the patterns in the activity signatures, we ask one subject to switch his activity from jumping to two micro activities, namely, sitting in a chair and phone typing, and finally, walking out of the room. The subject is asked to repeat the pattern twice to collect the corresponding range-doppler data under low and high doppler resolution. From the std in the heatmap across the entire activity time axis  <ref>, it is evident that low doppler resolution is adequate for capturing macro activities like walking and jumping. Still, typing and sitting does not have any significant signatures. On changing the radar configuration to high doppler resolution, we observe that micro activities like typing and sitting have better visibility. However, with this, the macro activities (walking, jumping) generate noisy data due to the higher resolution. Therefore, different doppler resolutions are crucial to capture the signatures corresponding to different activity types.  



  §.§.§ Multi-user Activity

Now, we elaborate on how to identify individual activity signatures when multiple subjects are of concern. It is worth noting that the captured range-doppler heatmaps often contain static clutters. Static clutters are any objects (walls, furniture, etc.) that are stationary but can reflect the mmWave signal and generate unwanted signatures in the range-doppler data. To elaborate on this, let us consider a scenario with two subjects, Subject 1 sitting and Subject 2 walking, inside the room, as shown in  <ref>. The room also contains multiple static clutters, such as wooden sheets and walls. From the range-doppler heatmap as shown in,  <ref> we observe multiple peaks at the range bins corresponding to the subjects and the static clutter. Here, we observe that the static clutter produces a higher magnitude along the zero Doppler axis, thereby signifying zero or no movement. On the other hand, the dynamic movements of the subjects are positioned across non-zero doppler bins. A major takeaway from the range-doppler heatmap is that static clutters are easily identifiable by their zero-doppler signatures.



 §.§ Benchmark Evaluation



The 2D range-Doppler heatmaps exhibit unique spatial patterns for each activity. However, the variations within each activity are confined to a limited range of bin regions, resulting in predominantly sparse data. To address this, we utilize a 2D convolutional Neural Network (2D-CNN) architecture to extract key features from these sparse heatmaps. The 2D convolution operation leverages the dependency of neighboring spatial values, and by concatenating multiple past frames in a sliding window fashion, we capture the temporal relationships over the past t frames.

For macro and micro activity feature extraction, we employ four and three 2D convolutional layers respectively, both with the `same' padding and ReLU activation. Following these layers, a global average pooling layer is used to extract the average spatial activation across the entire feature map. The feature extraction model is trained separately for macro and micro activities using 90% of the entire dataset. The remaining 10% of the dataset is used as the test set, where the final feature embeddings are obtained from the global average pooling layer. These embeddings are then visualized in a reduced 2-dimensional space using t-SNE.

As illustrated in  <ref> and  <ref>, the feature embeddings from the test dataset form distinct clusters for different activities. This clustering underscores the dataset's capability to effectively capture and differentiate between various human activity patterns. Additionally, when we compare this proposed classifier taking range-doppler heatmaps, it provides an accuracy of 98% and 95% for macro and micro activities, respectively. Whereas, the collected pointcloud data for the same set of activities when studied by a pointcloud-based HAR classifier, RadHAR <cit.>, we observe, although the classifier is performing well with for macro activities, it fails in the case of micro activities as pointcloud cannot capture human movements at a higher granularity.  The results are shown in  <ref> 





§ CONCLUSION

represents a significant advancement in the field of human activity recognition (HAR), offering a comprehensive and versatile resource for researchers and developers. By integrating range-Doppler heatmaps and point cloud data by employing adaptive doppler resolution, bridges the gap between macro and micro-scale activity recognition. This allows for the precise detection of both large and small movements, capturing the nuances of human behavior in diverse indoor settings. The dataset includes 19 different activities performed by multiple subjects, ensuring a rich variety of real-world scenarios and interactions. By making publicly available, we hope to foster further research and innovation in HAR, enabling the development of robust, accurate, and versatile systems for healthcare, security, and other uses.

	mmWaveMillimeter Wave
	RFRadio Frequency
	4G4^th Generation
	5G5^th Generation
	COTSCommercial off-the-shelf
	FMCWFrequency Modulated Continuous Wave
	ADCAnalog to Digital Conversion
	DBSCANDensity-Based Spatial Clustering of Applications with Noise
	EKFExtended Kalman Filter
	RKFRecursive Kalman Filter
	LoSLine of Sight
	NLoSNon Line of Sight
	FPSFrames Per Second

abbrv



§ APPENDIX FOR 



 §.§ Preliminaries

Before diving into the dataset[<https://github.com/arghasen10/mmdoppler> (Access: August 5, 2024)] documentation, we first discuss the preliminaries of mmWave sensing to familiarize with the features captured in the dataset, such as pointclouds and range-doppler heatmaps. The primary working principle of Commercial Off-The-Shelf (COTS) mmWave radars revolves around Frequency-Modulated Continuous Wave (FMCW) technology <cit.>. These radars emit continuous frequency chirps and perform a process called dechirping by mixing the transmitted signal (TX) with the reflected signal (RX) from objects. This creates an Intermediate Frequency (IF) signal. From this IF signal, we can derive (1) a Pointcloud, which is a discrete set of points representing the detected objects <cit.>, and (2) the Range-doppler heatmaps, which are 2D image representations where the abscissa represents the range (location of human subjects), and the ordinate indicates the doppler speed at which subjects are moving. These heatmaps combine range and doppler  information from radar signals, offering a detailed view of motion dynamics over time.



  §.§.§ Range-doppler Estimation

The distance between an object and the radar is determined by measuring the frequency difference between the reflected and transmitted signals <cit.>. This frequency difference, known as the beat frequency (f_b), occurs after a Round Trip Time (RTT) of τ. As shown in  <ref>, if T_C is the duration of the mmWave chirp across a bandwidth B, then the slope of the FMCW chirp is S = B/T_C = f_b/τ.

The RTT delay τ can be expressed as τ = 2d/c, where d is the distance to the detected object and c is the speed of light. Thus, the distance d can be calculated as d = c/2·T_C/B· f_b. To determine f_b, a Fast Fourier Transform (FFT), known as the range-FFT, is performed on the IF signal. This process produces frequency peaks at locations where reflecting objects are present, thereby estimating the range.

r5cm



    < g r a p h i c s >

    Working Principle of FMCW Radars.
    


To measure the velocity or doppler of a moving object, the radar transmits N chirps separated by a duration of T_C. If a subject moves at a velocity v, the phase difference between two successive RX chirps, corresponding to the motion vT_C, is given by Δϕ = 4π v T_C/λ. A second FFT, called doppler-FFT, is performed on these phase differences to determine the object's velocity. This data is captured in a 2D matrix called the range-doppler matrix 𝔻_D × R, where D and R are the numbers of doppler bins and range bins, respectively.



Equation (<ref>) shows the range-doppler matrix for R range bins and D doppler bins. The range-FFT captures objects at different locations:

    𝔻 =  [ 𝐏_1,1 𝐏_1,2     ⋯ 𝐏_1,R; 𝐏_2,1 𝐏_2,2     ⋯ 𝐏_2,R;     ⋮     ⋮     ⋱     ⋮; 𝐏_D,1 𝐏_D,2     ⋯ 𝐏_D,R ]_Doppler Bins×Range Bins

For each object, an angular FFT is performed across multiple doppler-FFTs.



 §.§ Pointcloud Estimation

The pointcloud is estimated using the standard Constant False Alarm Rate (CFAR) algorithm <cit.>, which detects peaks in the range-doppler matrix corresponding to detected objects. The pointcloud includes coordinates (x_i, y_i, z_i), doppler variations (d_i), and received power (p_i) of the detected objects. The pointcloud set S for N detected objects is given by S = ⋃_i=1^N (x_i, y_i, z_i, d_i, p_i). A sample snapshot of range-doppler heatmaps and pointcloud is shown in  <ref>.






 §.§ Dataset Documentation

Our dataset comprises a wide array of 19 different activity classes from Activities of Daily Living (ADLs), Instrumental Activities of Daily Living (IADLs) <cit.>, and daily indoor exercises: (i) macro activities like walking, running, jumping, clapping, lunges, squats, waving, vacuum cleaning, folding clothes, changing clothes, and (ii) micro activities like laptop-typing, phone-talking, phone-typing, sitting, playing guitar, eating food, combing hair, brushing teeth, and drinking water to ensure comprehensive coverage of daily human behaviors. For macro activities, since higher body movements are involved, a lower doppler resolution can easily capture those activities, while for micro activities, we leverage a higher doppler resolution as it involves subtle body movements. Below, we describe the data collection steps.


  §.§.§ Dataset Collection Steps

For data collection,  <cit.> mmWave radar is first connected to the PC via USB cable. One needs to install mmWave Demo Visualizer[<https://dev.ti.com/gallery/view/mmwave/mmWave_Demo_Visualizer/ver/2.1.0/>] from Texas Instruments as documented on our GitHub repository [<https://github.com/arghasen10/mmDoppler>]. Using this tool, the data gets saved on the local machine in a txt file in JSON format. We kept the saved file nomenclature as . We have collected the data by asking the subjects to perform the activities at different distances from the radar as well as with different orientations. Along with the raw sensor data collection, we connected a USB camera to the same device to collect video footage of the subject performing several activities. Due to privacy concerns, we have not uploaded the video footage of the subject performing the activities.



  §.§.§ Dataset Labeling

The raw datasets, stored in  format, are first converted into a single  DataFrame, with two additional columns: (i) subject name and (ii) activity name, which are derived from the corresponding filename. However, the filename alone does not ensure that the subject performed only that activity throughout the session. For instance, during a jumping activity session, the subject might take short breaks due to fatigue and resume the session after a 10-15 seconds gap.

To refine the coarse-grained activity annotations from the filename, we utilize video recordings from the session to obtain accurate activity annotations. The subjects are instructed to perform the activities at various orientations—left, right, front, and back—relative to the radar's bore-sight angle, as well as at different distances of 2 m, 3 m, and 5 m. These orientations and distances are meticulously annotated based on video recordings to ensure precise data labelling. These annotations, along with timestamps captured from the video, are stored in a  file. We then merge the raw DataFrame with the annotations from the CSV file to update the activity labels in the dataset.

We leave the annotations blank for periods where the subject is not performing any activities. Consequently, when merging the datasets, these unannotated entries are ignored in the final processed dataset. Thus, we only have the relevant data with accurate activity labels in the processed dataset.



  §.§.§ Feature Space Description

The collected and processed dataset consists of the following information: (i) datetime, the date and time when the data was recorded; (ii) pointcloud datasets; (iii) range-doppler heatmaps; (iv) Annotations of the position where the subject is performing the activity (2m, 3m or 5m) and orientation with respect to the radar (left, right, front, back), activity name such as (jumping or phone typing) and activity_class (whether macro or micro activity). These annotations are collected from the captured video frames with the help of three volunteers, who were given a remuneration of 20 USD for each 10-minute activity session. We have summarized the collected features in  <ref>. 





  §.§.§ Dataset Access and Metadata

Our dataset is distributed as an open-source dataset. We openly publish the dataset, along with the files for preprocessing, cleaning, and the codebase for the CNN activity classifier, along with the source code to process our data to run the existing pointcloud-based HAR activity classifier RadHAR <cit.>. Any work using our dataset should cite the main paper and the individual dataset sources listed in the documentation. The details of the codebase for data collection, pre-processing, and activity classification are provided in the GitHub repository: <https://github.com/arghasen10/mmDoppler>. 



 §.§ More classification report under different scenarios







Existing works on HAR in mmWave domain primarily focus on pointcloud based datasets, which can only capture coarse-grained movements such as walking, running or jumping <cit.>, thus these existing classifiers fail to capture micro activities having subtle body movements such as phone typing, eating, etc. so in this work, we have trained the classifier with the help of denser range-doppler heatmaps captured at different doppler resolution. We leverage a 2D-CNN classifier architecture to extract the key features for different activities. The classifier architecture is shown in  <ref>.

Finally, we compare the proposed activity classifier with the state-of-the-art pointcloud-based classifier RadHAR <cit.>. Since RadHAR captures 3D pointclouds, however, we have collected 2D pointclouds in this work, so we have modified[<https://github.com/arghasen10/mmDoppler/tree/main/models>] the RadHAR classifier so that it can incorporate 2D pointclouds for the classification task.




We have illustrated the confusion matrix for activity classification using the proposed classifier, trained on range-Doppler heatmaps, and RadHAR <cit.>, trained on point cloud datasets, in  <ref>. As depicted in  <ref> and  <ref>, the proposed 2D-CNN classifier effectively classifies activities using range-Doppler heatmaps. In contrast, the RadHAR classifier, trained on point cloud data, shows better performance in classifying macro activities involving significant movements (see  <ref>). However, it struggles with micro activities that involve subtle body movements (see  <ref>). This is primarily because existing methods and datasets are influenced by pointcloud datasets which although can work on macro activities but fail miserably on micro activities. Thus, our dataset containing range-Doppler heatmaps, with their denser representation, can capture features more accurately than the sparser point cloud representations.




