Paper Title: Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments.


















  The University of Queensland, CSIRO Data61
  Brisbane
  Australia
haodong.hong@uq.edu.au



  The University of Queensland
  Brisbane
  Australia
sen.wang@uq.edu.au



  The University of Queensland
  Brisbane
  Australia
helen.huang@uq.edu.au



  The University of Adelaide
  Adelaide
  Australia
qi.wu01@adelaide.edu.au


Corresponding author

  CSIRO Data61, The University of Queensland
  Brisbane
  Australia
jiajun.liu@csiro.au











§ ABSTRACT 
Real-world navigation often involves dealing with unexpected obstructions such as closed doors, moved objects, and unpredictable entities. 
However, mainstream Vision-and-Language Navigation (VLN) tasks typically assume instructions perfectly align with the fixed and predefined navigation graphs without any obstructions.
This assumption overlooks potential discrepancies in actual navigation graphs and given instructions, which can cause major failures for both indoor and outdoor agents.
To address this issue, we integrate diverse obstructions into the R2R dataset by modifying both the navigation graphs and visual observations, introducing an innovative dataset and task, R2R with UNexpected Obstructions (R2R-UNO).
R2R-UNO contains various types and numbers of path obstructions to generate instruction-reality mismatches for VLN research.
Experiments on R2R-UNO reveal that state-of-the-art VLN methods inevitably encounter significant challenges when facing such mismatches, indicating that they rigidly follow instructions rather than navigate adaptively. 
Therefore, we propose a novel method called ObVLN (Obstructed VLN), which includes a curriculum training strategy and virtual graph construction to help agents effectively adapt to obstructed environments. 
Empirical results show that ObVLN not only maintains robust performance in unobstructed scenarios but also achieves a substantial performance advantage with unexpected obstructions.   
The source code is available at <https://github.com/honghd16/ObstructedVLN>.







<ccs2012>
<concept>
<concept_id>10010147.10010178.10010224.10010225</concept_id>
<concept_desc>Computing methodologies Computer vision tasks</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10002951.10003227.10003251</concept_id>
<concept_desc>Information systems Multimedia information systems</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>


[500]Computing methodologies Computer vision tasks
[500]Information systems Multimedia information systems
























Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments
    Jiajun Liu
    August 5, 2024
=========================================================================================





§ INTRODUCTION






Vision-and-Language Navigation (VLN) <cit.> requires agents to follow natural language instructions to reach a specified destination.
Recently, there has been a growing interest in this area due to its great potential for real-world applications such as household robots, especially with the booming of Large Language Models (LLMs) <cit.>.
However, current VLN tasks are limited by several unrealistic assumptions, making their application remain in simulators with few real-world robotic deployments <cit.>.


One significant constraint is what we call the “perfect instruction assumption”, which implies that instructions are always perfectly aligned with the environment, neglecting real-time dynamics like unexpected obstructions.
Agents trained under this assumption excel in following instructions but lack adaptability for actual navigation where discrepancies between instructions and reality are commonly seen.
For example, as shown in <ref>, the human instructs the agent to “Walk straight down the hall” based on prior knowledge of the house. 
However, real-world environments have changed, with unforeseen obstacles like a suitcase blocking the hallway.
While humans can quickly adapt and find detours, current VLN agents struggle with these instruction-reality mismatches, often leading to navigation failures.


To address this issue, it is essential to introduce these discrepancies into VLN.
While various factors can lead to such discrepancies, this work focuses on one of the most representative and prevalent causes: obstructions. 
We propose to integrate obstructions into existing discrete VLN environments to block the path described by the instruction, resulting in an instruction-reality mismatch.
Notably, although both approaches involve obstructions, our work differs from previous work on obstacle avoidance <cit.> in terms of problem setup, navigation focus, and potential solutions (See <ref>), leading us to choose the discrete setting.




Therefore, we make various modifications to the navigation graphs and visual observations of the R2R dataset <cit.>, proposing the R2R with UNexpected Obstructions (R2R-UNO) dataset as the first VLN task to emphasize the instruction-reality mismatches.
At the graph level, we selectively block edges whose removal does not impact the overall graph connectivity in existing paths, ensuring agents can still reach the destination.
To keep visual coherence with graphs, we design an object insertion module that leverages text-to-image inpainting techniques <cit.> to integrate diverse objects into different scenes seamlessly.
Due to the instability of inpainting results, a filtering module is employed to select the high-quality ones from multiple candidates to ensure obstruction generation.
Our study using R2R-UNO reveals that state-of-the-art VLN agents <cit.> perform poorly in obstructed environments, limiting their practical applications.

Although we expect agents to perform well with mismatches, it is crucial to maintain their performance in original environments simultaneously, which represent most situations. 
However, directly training with these two types of data poses challenges for agents to learn beyond following instructions and to differentiate between mismatched and aligned scenarios. 
Empirical results show that discrepancies between these two environments can lead to imbalanced optimization, favoring one type. 
To address this, we develop the Obstructed VLN (ObVLN) method, including a curriculum training strategy <cit.> to organize the training and a novel graph construction mechanism to introduce virtual nodes for blocked edges to facilitate efficient exploration.
The overall framework including data generation and agent training is presented in <ref>.


We conduct comprehensive experiments on R2R, REVERIE <cit.>, and R2R-UNO datasets to demonstrate the significance of incorporating instruction-reality mismatches in VLN. 
Established methods, such as DUET <cit.>, struggle in obstructed settings, with a significant 30% drop in Success Rate (SR). 
The proposed object insertion and filtering modules provide crucial visual feedback aligned with navigation graph changes, proved by the improved performance of agents using inpainting images.
By employing ObVLN, agents not only maintain robust performance in original scenarios but also effectively adapt to mismatches, achieving an impressive 67% SR, marking a significant advancement.
We summarize our contributions in this paper as follows: 


  * We address the underexplored issue of instruction-reality mismatches in VLN by proposing R2R-UNO, the first VLN dataset that includes such mismatches through graph changes and diverse obstruction generation, offering a unique challenge that reflects real-world navigation complexities.

  * We highlight the lack of adaptability in current VLN methods for obstructed environments and propose ObVLN as a solution, which employs curriculum learning and virtual graph construction to enhance agent adaptability.

  * Through extensive experiments, we prove the significance of introducing R2R-UNO in VLN research and show that ObVLN performs well in both original and obstructed environments, achieving a significant 23% SR increase in R2R-UNO.
 




§ RELATED WORK

Vision-and-Language Navigation
In VLN, agents need to navigate within simulated environments like Matterport3D <cit.> following natural language instructions. 
Numerous methods <cit.> and datasets <cit.> have been developed to address various challenges in this domain.
Some works address the data scarcity problem by introducing extra sources, including synthesized instructions <cit.>, additional environments <cit.>, and predicted scenes <cit.>.
Recently, ScaleVLN <cit.> synthesized enormous high-quality instruction-trajectory pairs for HM3D <cit.> and Gibson <cit.> environments to make agent performance approach human results.
Other works <cit.> apply diverse network structures such as LSTM <cit.>, Transformer <cit.>, and Graph Neural Networks <cit.> to enhance cross-model alignment and decision-making ability.
For example, HAMT <cit.> encodes the full history information through transformers and integrates it with instructions and observations for better action prediction.
DUET <cit.> further maintains a topological map to enable agents aware of global visual representations to make global decisions instead of adjacent viewpoints.
We adopt HAMT and DUET as the evaluation models due to their superior performance and difference in whether map-based.

 Environment Changes in VLN
Many works <cit.> have proposed to modify VLN environments for different purposes, which can generally be categorized based on the changing level. 
One category involves changing the visual observations to augment data for improving agent generalization.
For example, Li et al. <cit.> utilized image captioning to obtain language descriptions of panoramas and employed generative models to produce novel views based on these descriptions.
Conversely, another category focuses on graph-level adjustments rather than the visual level to go beyond the established navigation graphs by mixing up different graphs <cit.> or substituting a fixed viewpoint with proximate locations <cit.>.
Notably, VLN-CE <cit.> abandons the graph-based navigation paradigm and allows agents to traverse continuous environments freely to enhance task realism.
Differently, our task makes changes at both visual and graph levels to block edges in the graph and generate obstacles in the views to generate instruction-reality mismatches.

 Obstacle Avoidance

Obstacle avoidance has been a long-standing research challenge in visual navigation <cit.>.
In VLN, ETPNav <cit.> applies an obstacle-avoiding controller with a trial-and-error heuristic to explicitly escape from deadlocks.
SafeVLN <cit.> employs a LiDAR-based waypoint predictor and a re-selection strategy to avoid non-navigable waypoints and obstacles. 
However, our work diverges from these works under the continual setting <cit.> in three aspects, leading us to choose the discrete setting.
Firstly, we target a broader range of situations with instruction-reality mismatches, including avoidable obstacles, closed doors, rearranged furniture, instruction errors, etc.
These scenarios can be effectively captured by graph changes but are hard to model as continuous signals.
Secondly, while obstruction avoidance focuses on assessing the properties of obstructions to avoid them, our work emphasizes changes in navigation graphs that render the instructions temporarily invalid, ignoring the nature of the obstructions.
The discrete setting decouples path planning from technicalities, such as the shape and size of obstacles, thereby focusing on high-level adaptability when instructions fail and enhancing its generality.
Finally, addressing instruction-reality mismatches involves more than just navigating around obstructions. 
It requires a complex strategy to find detours and the ability to navigate without instruction guidance, posing a significant adaptability challenge which is too hard for current continuous agents.

 Object Insertion
Inserting novel objects into target images has always been a research challenge in Computer Vision.
Early works <cit.> employ the cut-and-paste strategy to merge two pictures, which is straightforward but lacks photorealism.
With the development of neural networks, object insertion can be achieved in various ways and higher quality, including image synthesis <cit.>, neural radiance fields editing <cit.>, image generation <cit.>, etc.
In navigation, THDA <cit.> inserts 3D scans of household objects into random locations as navigation goals to augment the training data for ObjectGoal Navigation <cit.>.
In VLN, Envedit <cit.> leverages semantic image synthesis <cit.> to create objects based on modified environment semantics as augmented environments.
In this work, we uniquely take advantage of the text-to-image inpainting models <cit.> to seamlessly embed desired objects within specific view locations without influencing visual surroundings.



§ OBSTRUCTED ENVIRONMENTS



In this section, we first introduce the problem formulation of VLN on vanilla and obstructed environments, and then describe our proposed R2R-UNO dataset.



 §.§ Problem Setup
 
In VLN, the agent must follow a natural language instruction x̂=(x_1,x_2,...,x_L) with L words to navigate a simulated environment.
This environment is usually discrete with a predefined undirected navigation graph 𝒢={𝒱, ℰ} with navigable nodes 𝒱 and connectivity edges ℰ.
At each timestep t, the agent perceives a panoramic representation, comprised of N=36 views O_t={o_t^i}_i=1^N and an orientation feature that encodes heading θ and elevation ϕ information of its current node v_t. 
It then determines an action a_t to transition to one of the neighboring nodes 𝒩(v_t) by selecting the view that aligns best with the target node from the current panorama O_t.
An additional “STOP” action is available to conclude the navigation process.
For each instruction x̂, there exists a corresponding ground truth path P=<v_1, e_1, v_2, e_2, ⋯, v_n> representing the intended trajectory with n nodes for the agent. 
However, previous work presumes perfect alignment between instruction and reality, assuming that all edges <e_1, e_2, ⋯, e_n-1> are consistently accessible.
This assumption overlooks the dynamics of real-world navigation, where the graph 𝒢 may have changed.
Among various reasons for graph changes, we focus on the most representative one: obstructions.
In our obstructed environments, some specific edges E_s ⊆ P might be obstructed, making parts of the instruction inapplicable to the current scenario. 
This obstruction also leads to corresponding visual changes in the panoramic views O_t of nodes linked by these edges. 
When meeting obstructed edges E_s, the agent must find a substitute path P' to reach the final destination v_n.



 §.§ R2R-UNO



In this section, we introduce the proposed R2R-UNO dataset in detail, including the graph changes and visual modifications.





  §.§.§ Graph Changes

For a path P=<v_1, e_1, v_2, e_2, ⋯, v_n> in R2R, we first identify all redundant edges within this path, denoted as E_r ⊆ P.
A redundant edge e_i=(v_i, v_i+1) ∈ E_r is defined by the property that its removal does not affect the overall connectivity of the graph, ensuring an alternate path P' between v_i and v_i+1. 
To account for scenarios where multiple edges are concurrently obstructed, we assess the collective redundancy of combinations of these redundant edges E_r and categorize these combinations by the number of obstructed edges x into Block-x set.
Since R2R paths have 4 to 7 nodes in length, we set N_max=3 as the maximum number of obstructed edges, generating three sets Block-1,2,3, each containing individual training and evaluation splits.

With a Block-x set, consider a combination of redundant edges C=<e_i_1, e_i_2, ⋯, e_i_x>.
For each edge, we identify the shortest path <P'_1,P'_2,⋯,P'_x> between the corresponding nodes in the modified graph 𝒢={𝒱, ℰ-C} to replace C in the original path P, generating the path P as the new intended path for agents:

    P̅ = ⟨ v_1, …, v_i_j, P'_j, v_i_j+1, …, v_n ⟩,   ∀ j ∈1, 2, …, x

We refer to P̅ as the “real path” to be traversed by agents, in contrast to the “instructional path” P based on the navigation instructions x̂.
Therefore, the instruction-reality mismatch is defined as P≠ P, which demands agents actively seek alternate paths. 
To exclude excessively lengthy and impractical new routes, each Block-x set possesses a “real path” length restriction as L(P̅) ≤ 10+5 · x. 
As a result, 98.8% R2R paths are modified to be one or more new paths in R2R-UNO.
As indicated in <ref>, the R2R-UNO sets contain a greater number and length of paths than R2R, with the average path length increasing with x, denoting greater navigation difficulty.

Notably, when constructing P̅, we deliberately avoid shortening the path when P' contains future viewpoints from P, even if this leads to the loop formation in P̅.
This design aligns with practical navigation scenarios since when deviating from the instruction, it is challenging for agents to align the current viewpoint with the described path due to the absence of contextual clues. 
Therefore, it is logical and efficient for agents to seek a detour around the obstruction before attempting to re-align with the instructions.



  §.§.§ Visual Changes


To align with the graph changes, we introduce two novel modules to infuse various objects into panoramic views of nodes along redundant edges, as shown in <ref>.

The first one is the object insertion module, which employs a stable diffusion inpainting model <cit.> to approach the issue from an inpainting perspective.
Consider a redundant edge e linking nodes v_a and v_b. 
We elaborate the process of modifying the panoramic view O_a={o_a^i}_i=1^36 at node v_a as follows, which is similarly applied to O_b.
First, we localize the other node v_b within O_a to find the corresponding discrete view o_a^b and calculate the pixel coordinates (x_b, y_b) in o_a^b as follows:

    f    = H / (2 ·tan(δ / 2)) 
    
    x_b    = tan(θ_r) · f + W/2 
    
    y_b    = tan(ϕ_r) · f + H/2

W, H is the width and height of o_a^b, while θ_r and ϕ_r is the relative heading and elevation from v_a to v_b, and δ is the vertical Field of View (FoV) to calculate the focal length f.
A right trapezoid mask M_a^b is then generated, expanding from the bottom edge to around the point v_b with a specified width to cover potential areas of paths from v_a to v_b.
This is based on the observation that paths in these views either stretch from the bottom middle to the target point (x_b, y_b) or form a vertical line from the bottom to the target. 
Finally, we combine an object name obj with a predefined prompt template to formulate the final prompts pr, which, along with the view image o_a^b and mask M_a^b, serve as the input to the inpainting model to generate the inpainted view ô_a^b:

    ô_a^b = Inpainting(o_a^b, M_a^b, pr)


While inpainting models can generate visually coherent and photo-realistic images, their instability is notable, with a certain probability of failing to incorporate the desired object.
This instability often arises when the area outside the mask already contains elements similar to the intended object, such as a “chair” or “table”, which can mislead the inpainting model even with intricate prompts.
Therefore, we propose a filtering module to improve the quality of inpainted images, which evaluates multiple generated candidates with different objects to filter out higher-quality ones.
Specifically, to enrich the generated scenarios, we first carefully select ten object categories as obstructions based on their frequencies in both the Matterport3D dataset and real-world environments, including chair, table, sofa, potted plant, basket, exercise equipment, vacuum cleaner, suitcase, toy, and dog. 
We then utilize the object insertion module to produce a set of novel views as candidates, each with a different kind of object.
To reflect the inpainting quality, we employ the CLIP <cit.> model to assess the compatibility score s_c of each view-object pair.
Note that this assessment considers only the modified part of the view by combining the mask M_a^b:

    s_c = CLIP(ô_a^b ∘ M_a^b, obj)

The ∘ operator represents pixel-wise multiplication.
These scores are aggregated to form a dataset including all redundant edges within R2R. 
Visual analyses of this dataset indicate that the score distribution for each object category often takes the shape of a bimodal Gaussian distribution, with one peak 𝒩(s_c | μ_1, σ_1^2) for successful incorporation and another 𝒩(s_c | μ_2, σ_2^2) for failure:

    P(s_c) = π_1 𝒩(s_c | μ_1, σ_1^2) + π_2 𝒩(s_c | μ_2, σ_2^2)

π is the mixing coefficient, 𝒩(·| μ, σ^2) represents a Gaussian distribution with mean μ and variance σ.
Based on this insight, we train a bimodal Gaussian Mixture Model (GMM) <cit.> for each category and use them to decide which images are qualified: 

    q =  
    1    if 𝒩(s_c | μ_1, σ_1^2) > 𝒩(s_c | μ_2, σ_2^2) 
    
    0    otherwise

Finally, we randomly select one of those qualified candidates as the modified view, or, in the absence of suitable options, the choice is narrowed to the top three scorers.
This module not only largely enhances the probability of incorporating visible obstructions but also ensures the selection of contextually fitting objects.
We present the distributions of object categories and their compatibility scores in the appendix to prove the diversity of R2R-UNO.

All steps above only concern one discrete view o_a^b.
The final step is propagating the updated view ô_a^b to adjacent views o_a^j_1:8 containing overlapped areas with o_a^b to maintain consistency across the panorama.
We first calculate the transformation matrix K_i for view o_a^j_i based on SIFT features <cit.> matching.
We then apply K_i to project the inpainted part of ô_a^b onto the adjacent view and merge it with the original view to construct a coherent novel panorama:

    K_i = MATCH(SIFT(o_a^b), SIFT(o_a^j_i)) 
       ô_a^j_i = o_a^j_i· (1 - K_i * M_a^b) + K_i * ô_a^b · (K_i * M_a^b)

Here, * denotes the projection operation by pixel-wise matrix multiplication applied to image coordinates.

It is important to note that we only perform 2D inpainting for nodes linked by redundant edges, which may lead to multi-view inconsistency.
Although 3D object insertion can alleviate this problem, we stick to the 2D method for two reasons.
Firstly, current 3D techniques <cit.> mainly rely on limited pre-built object datasets <cit.>, which lack diversity and can cause visual inconsistencies in aspects like lighting and style. 
Other generative 3D methods <cit.> struggle to generate high-resolution photo-realistic images and often produce noticeable artifacts.
Secondly, the impact of multi-view inconsistency is minimal in our task, as instruction-reality mismatches are primarily defined by graph changes.  
Agents can only detect graph changes when moving to nodes alongside redundant edges, making visual modifications at other nodes irrelevant since agents should still follow instructions without detecting the mismatch.
Moreover, various environment augmentation methods <cit.> have proved that this inconsistency would not affect agent performance in real-world scenarios.



§ INSTRUCTION-REALITY MISMATCHES AND SOLUTION


In this section, we show that current VLN methods perform poorly when encountering instruction-reality mismatches and thus propose the ObVLN method to solve this problem.





 §.§ Current VLN Methods Struggle in R2R-UNO



To assess the impact of instruction-reality mismatches, we conduct zero-shot evaluations on the R2R-UNO validation unseen splits using five advanced VLN methods, known for their excellent performance under the perfect instruction assumption, including RecBERT <cit.>, HAMT <cit.>, DUET <cit.>, GELA <cit.> and ScaleVLN <cit.>.
<ref> illustrates how their success rates vary with different numbers of blocked edges.
While most methods achieve a 60%–70% success rate in original environments, the curves sharply decline to around 40% when a single edge is obstructed, with further deterioration observed in the Block-2 and Block-3 scenarios. 
ScaleVLN outperforms other methods in R2R-UNO owing to its strong generalization ability brought by extensive training data, but it still suffers from a nearly 20% performance reduction in the block-1 set of R2R-UNO.
We conclude that current VLN models are overly focused on instruction-following capabilities and lack essential basic navigation functionality to adapt to graph changes.
These findings emphasize the severity of using the perfect instruction assumption in VLN research and the urgency for its resolution.





 §.§ ObVLN

With R2R-UNO, agents can be trained in obstructed environments to better adapt to instruction-reality mismatches.
However, significant gaps exist between original and obstructed environments, such as navigation requirements, reliance on instructions, and trajectory numbers, making direct training on these two types of data compromise the performance in both environments (See <ref>).

Therefore, we propose Obstructed VLN (ObVLN), including a novel training strategy and a graph construction mechanism to organize the training and help agents deal with obstructions.
We advocate a curriculum learning strategy to treat original and obstructed environments as distinct yet complementary tasks to facilitate the training.
This strategy leverages the instruction-following skills gained in original environments as a foundation to tackle obstructed settings.
Initially, agents are trained in purely unobstructed environments.
As training progresses, the sample ratio of obstructed environments α gradually increases until it reaches a predefined maximum α_max at step c, defined as follows:

    α(t) = min(α_max, t/c·α_max)

where t is the current training step.
All training batches include instances from both settings to ensure smooth optimization.

To facilitate exploration for detours, we introduce a graph construction mechanism for graph-based methods to incorporate “virtual nodes” to represent currently inaccessible nodes due to obstructions.
This mechanism consists of three steps, with the pseudo-code in the appendix.
Firstly, when encountering an obstacle at node v_a, we estimate the location of the obstructed node v_b based on the direction of the obstructed view and a predefined distance d (3 meters in R2R-UNO).
This estimation allows us to conceptualize a virtual node v^*_b within the topological graph as a placeholder for v_b. 
Next, as the agent moves, it consistently computes its distances to v^*_b and v_a, denoted as D(v^*_b) and D(v_a), respectively. 
This distance computation is crucial for determining the proximity of the agent to the virtual and obstructed nodes.
Finally, if the distances satisfy D(v^*_b) < θ and D(v_a) < θ for a given threshold θ, and the current view indicates obstruction, we infer that the agent has arrived or moved close to v_b. 
So we connect the virtual node v^*_b and current node v_c with a zero-weight edge, indicating their alignment.
This mechanism helps agents to seek alternative routes more purposefully and increase exploration efficiency.



§ EXPERIMENTS




 §.§ Datasets and Evaluation Metrics

We mainly evaluate our methods on the widely used VLN benchmark R2R <cit.> with step-by-step instructions and our proposed R2R-UNO to focus on the challenge of instruction-reality mismatches. 
We present results on the goal-oriented benchmark REVERIE <cit.> without such mismatches in the appendix.
R2R is built on the Matterport3D dataset <cit.>, including 10,800 panoramic views from 90 building-scale scenes.
Each R2R path has three or four natural language instructions from human annotators.
R2R has four splits: train, validation seen (val seen), validation unseen (val unseen), and test unseen.
We utilize the train split for training and the val seen and val unseen splits for evaluation to align with R2R-UNO.

For evaluation, we follow previous works <cit.> to use four primary metrics in VLN: 
(1) Trajectory Length (TL): the total navigation length in meters; 
(2) Navigation Error (NE): the distance between the stop location and the target; 
(3) Success Rate (SR): the ratio of agents stopping within 3 meters of the target; 
(4) Success rate weighted by Path Length (SPL) <cit.>: SR normalized by the ratio between the length of the shortest path and the predicted path. 
Metrics related to instruction fidelity like CLS <cit.> or nDTW <cit.> are not included due to the modified trajectory with obstacles.



 §.§ Implementation Details


For R2R-UNO creation, we use the stable-diffusion-v1.5-inpainting model for object insertion and the CLIP ViT-L/14 to evaluate text-image pairs.
We use brute force and K-Nearest Neighbors matching to align the SIFT features of two adjacent views.
For methods in <ref>, we use their best models on the validation unseen split and extract the features of the obstructed environments according to their settings.
We employ HAMT <cit.> and DUET <cit.> for navigation training and follow the implementation details in their official repositories.
The obstructed environments are only used in the fine-tuning stage.
The maximum sample ratio α_max is set to 0.5 with the increasing step c as 20,000.
We increase the maximum action length to 30 for longer ground truth paths.
PREVALENT <cit.> is used as the augmented data to stabilize the training.
We utilize the AdamW optimizer <cit.> and select the best model based on the average SPL in the val unseen splits of R2R and R2R-UNO.
All models are fine-tuned for 100K iterations with a learning rate of 1e-5 and a batch size of 8 on a single NVIDIA A6000 GPU.



 §.§ Main Results

We first demonstrate that our proposed obstructed environments and ObVLN can facilitate agent adaptation to instruction-reality mismatches.
Therefore, we apply HAMT and DUET to three distinct training settings:
(1) Basic: training with R2R; 
(2) +OE: incorporating data from both the R2R and R2R-UNO;
(3) +ObVLN: using the ObVLN method on R2R and R2R-UNO.
Note that both settings 2 and 3 include all three sets of R2R-UNO.
<ref> presents the navigation performance of different models on the val seen and unseen splits of R2R and R2R-UNO.
The results show that models trained with two types of data significantly outperform those trained only on R2R in navigating obstructed scenes within the R2R-UNO dataset.
However, this improvement is accompanied by a noticeable degradation in R2R performance, like the 6% and 7% SR drop of DUET in the seen and unseen splits, respectively.
This phenomenon can be explained by the considerable TL increase in R2R, suggesting that agents are over-optimized for obstructed environments and tend to take detours even without obstructions.
Our ObVLN method effectively addresses this issue.
For HAMT, ObVLN surpasses the basic setting in R2R, achieving the best results across all four sets of R2R and R2R-UNO.
For DUET, the topological map and global action space lead agents to return to the node with the blocked edge which aligns best with the instruction.
This results in a slight loss of efficiency in terms of SPL in R2R for ObVLN.
However, it still achieves a comparable SR to the basic setting and significantly outperforms setting 2 in R2R.
Additionally, it achieves state-of-the-art results in all three sets of R2R-UNO.
The advantage of HAMT with ObVLN is less significant because HAMT is not a map-based approach, making our graph construction mechanism inapplicable.

While DUET outperforms HAMT by a 7% SR increment on R2R, this superiority becomes much more significant in obstructed environments, in which DUET achieves around 25% SR lead in the most challenging Block-3 set.
We attribute this to the topological map design in DUET, which significantly enhances exploration efficiency and serves as a critical component in finding detours. 
This advantage is consistent with the large lead of DUET on exploration-intensive, object-oriented datasets like REVERIE <cit.>.



 §.§ Ablation Study

In this section, we conduct an ablation study on several important components to show their effectiveness.







  §.§.§ Two Modules for R2R-UNO

We first evaluate the impact of the Object insertion Module (OM) and the Filtering Module (FM) by generating different Block-1 sets for training and evaluation.
Without these two modules, the obstructed environments only contain graph changes without visual modifications.
<ref> presents the performance of HAMT and DUET on the val unseen splits using different modules.
The R2R performance is consistent across all experiments due to ObVLN. 
For obstructed environments, using the object insertion module to modify the panoramic views can enhance agent navigation by providing critical visual feedback on graph changes, evidenced by performance gains.
Moreover, including the filtering module further improves the inpainting quality, leading to more successful obstruction generations and the best performance.



  §.§.§ Sampling Strategy

We explore four sampling strategies for DUET to utilize both original and obstructed environments:
(1) Path-wise, where a path is randomly selected from the combined pool of R2R and R2R-UNO paths;
(2) Task-wise, where data is sampled from original and obstructed environments in a Bernoulli mixture distribution with probabilities 1-α_t and α_t, respectively;
(3) Instruction-wise, where for each instruction, one path is randomly chosen from all possible paths.
(4) Curriculum sample (Ours), which gradually increases the sampling ratio α up to α_max by step c.
<ref> shows their performances on the the val unseen splits of R2R and R2R-UNO.
We set α_t=0.5 to match α_max and provide experiments on different α_t and α_max in the appendix.
Among all strategies, our curriculum sampling achieves dual superiority, achieving the best performance in both R2R and R2R-UNO.
Other methods experience performance drops on R2R due to the over-optimization problem.
Path-wise sampling obtains the worst performance due to path imbalance, with more training on paths with many redundant edges while ignoring those with none or less.







  §.§.§ Graph Construction Mechanism

We compare our graph construction mechanism with two baselines: the Vanilla setting that overlooks the obstruction in the graph and an idealized Oracle setting that receives the ground truth information about nodes occluded by obstructions as a performance ceiling.
<ref> shows the performance of DUET with different graph construction methods on the val unseen splits of R2R-UNO.
Our approach, incorporating virtual nodes into the graph, significantly outperforms the vanilla setting on all three sets, especially in more challenging scenarios with three obstructed edges.
As expected, the Oracle graph achieves the highest performance due to its access to accurate, unobstructed node information. 
These findings emphasize the necessity of equipping agents with obstruction-aware capabilities and the benefits of integrating obstructions into the topological map.




 §.§ Qualitative Analysis

We present some obstructed environments from R2R-UNO in <ref> comparing with corresponding original views and those generated without the filtering module. 
Our method successfully integrates various objects into specific locations within the original views, creating realistic and contextually harmonious obstructions. 
In contrast, results without the filtering module often fail to include the objects, demonstrating the critical role of this module in enhancing the inpainting reliability.
Additionally, we evaluate the generation quality using the CLIP <cit.> score and present the scores for each category in <ref>.
The filtering module consistently improves and stabilizes the generation quality, achieving higher scores and lower variance across all categories,  aligned with the examples.



§ CONCLUSION

This work introduces obstructed environments into VLN to address the prevalent issue of instruction-reality mismatches in real-world navigation. 
We present R2R-UNO, the first VLN dataset to incorporate such mismatches by integrating diverse obstructions into R2R at both the graph and visual levels through a novel object insertion and filtering module.
Using R2R-UNO, we demonstrate that current VLN methods struggle in obstructed settings and further propose the ObVLN method to help agents effectively adapt to obstructed environments.
Through these innovations, our agents achieve state-of-the-art results in R2R-UNO while maintaining robust performance in R2R.
We believe addressing the perfect instruction assumption is crucial for the practical application of VLN agents and for assessing their adaptability to navigate beyond only following instructions.







This work is supported by projects DE200101610, DP230101753 funded by Australian Research Council, and CSIRO's Science Leader project R-91559.





ACM-Reference-Format






































 Paper Title: Synthetic Simplicity: Unveiling Bias in Medical Data Augmentation.












Indian Institute of Technology Madras, India 
 GE HealthCare, Bangalore, India
	
	

Krishan Agyakari Raja Babu et al.



Synthetic Simplicity: Unveiling Bias in Medical Data Augmentation
    Krishan Agyakari Raja Babu1
 Rachana Sathish2Mrunal Pattanaik2
Rahul Venkataramani2{rahul.venkataramani}@gehealthcare.com
    
=============================================================================================================================



§ ABSTRACT 
Synthetic data is becoming increasingly integral in data-scarce fields such as medical imaging, serving as a substitute for real data. However, its inherent statistical characteristics can significantly impact downstream tasks, potentially compromising deployment performance. In this study, we empirically investigate this issue and uncover a critical phenomenon: downstream neural networks often exploit spurious distinctions between real and synthetic data when there is a strong correlation between the data source and the task label. This exploitation manifests as simplicity bias, where models overly rely on superficial features rather than genuine task-related complexities. Through principled experiments, we demonstrate that the source of data (real vs. synthetic) can introduce spurious correlating factors leading to poor performance during deployment when the correlation is absent. We first demonstrate this vulnerability on a  digit classification task, where the model spuriously utilizes the source of data instead of the digit to provide an inference. We provide further evidence of this phenomenon in a medical imaging problem related to cardiac view classification in echocardiograms, particularly distinguishing between 2-chamber and 4-chamber views. Given the increasing role of utilizing synthetic datasets, we hope that our experiments serve as effective guidelines for the utilization of synthetic datasets in model training.









§ INTRODUCTION


The acute challenge of data scarcity in medical imaging has spurred the development and application of various methods, including transfer learning, self-supervised learning, and data augmentation. More recently, synthetic data, either alone or combined with real data, has emerged as a popular technique to address the inherent data scarcity in medical imaging. The growing prominence of synthetic data is largely attributed to the efficiency of advanced generative techniques, such as generative adversarial networks (GANs), diffusion models  <cit.>, which produce high-quality visual images with relatively straightforward training. As visual generative models continue to improve, researchers are increasingly exploring the utility of generated synthetic samples to augment real data  <cit.>, aiming to achieve fairness and generalization.

Since the images produced by generative models appear extremely realistic, deep learning practitioners have increasingly adopted the use of synthetic data as a substitute of real data. In this paper, we investigate whether synthetic images possess statistical signatures, which may perhaps be inconspicuous visually, specific to their generation algorithms, distinguishing them from those found in real images. These distinctive features could potentially be exploited by downstream algorithms. It is well-known that discriminative deep learning methods tend to exploit the simplest features while disregarding other relevant aspects—a phenomenon referred to as shortcut learning due to “simplicity bias”  <cit.>. Medical imaging is particularly prone to these scenarios as data for a particular class could be hard to obtain (e.g. pathologies). Since the presence of “simplicity bias" can lead to sub-optimal performance during deployment, it is necessary for model developers to be cognizant of the perils of indiscriminate use of synthetic data during model development.

In summary, our contributions are as follows:

	
  * We conceive the possibility of high correlations between source of data and target resulting in deep learning model exploiting non-task specific features leading to poor performance during deployment.
	
  * We expose, with experiments on controlled datasets covering both natural images (digit classification) and medical imaging tasks (echocardiograph view classification), the problem of models exploiting the source of dataset (real vs. synthetic) as opposed to the task-specific causal feature.






§ RELATED WORK


Diffusion Probabilistic Models (DDPM),  <cit.>, have demonstrated remarkable performance in generating high-quality image samples. DDPMs leverage iterative denoising processes to model complex data distributions effectively. The core idea of DDPM revolves around iteratively applying a denoising process to generate samples from a target distribution. To enhance control over semantic details, recent advancements in generative modeling have focused on conditioning models on object-level segmentation masks. A semantic diffusion model (SDM) was proposed in a recent study  <cit.> for generating synthetic 2D echocardiograms. In this study, we adopt the SDM framework as our model for generating synthetic images.


The promise shown by DDPMs, SDMs and similar models have spurred the utilization of synthetic data  to enrich the training dataset <cit.>, anomaly detection <cit.>, super resolution <cit.> etc.
Due to the increasing utilization of synthetic datasets in medical imaging, it is imperative to critically analyze the potential pitfalls carefully. Particularly, the phenomenon of simplicity bias has been observed in discriminative deep learning methods, where models tend to exploit spurious, low-level statistical patterns that correlate significantly with the target rather than high-level semantic features, leading to poor out-of-distribution (OOD) performance . This issue is exacerbated in scenarios where synthetic data is used to augment real datasets. Particularly, three implications of the simplicity bias have been discussed by <cit.>—lack of robustness, lack of reliable confidence estimates, and suboptimal generalization.  While the dangers of shortcut learning in the context of healthcare <cit.> have been discussed, there is no prior work in the context of spurious features manifested due to utilization of synthetic data.




§ METHODOLOGY


In order to explore the manifestation of synthetic simplicity in medical data augmentation, we consider the binary classification problem on a dataset D = {(X_1, y_1),(X_2, y_2),…,(X_N, y_N)} comprising of N samples, where input images X∈ℝ^H × W and labels y ∈{-1, +1}.Each image can be considered to possess M latent features represented as f ∈ℝ^M which can be leveraged for classification. One of the features can be assumed to be a binary feature f_k ∈{0, 1}, where k denotes the feature index, representing whether a given sample is real or generated synthetically. We intentionally set the correlation between this binary feature and the output label to be high during training and evaluate during the testing phase when the correlation is lower. The test set consists of samples of the form (X, y = 1, f_k = 0), indicating the low correlation between y=1 and f_k=0. We vary the degree of correlation using the parameter α to explore different scenarios of synthetic simplicity.

So, with this configuration, the model during training has the potential to learn the simpler but spurious feature of real vs. synthetic. Since the correlation between the task label and data source is quite high, this will result in excellent performance during training. However, during evaluation when the correlation is lower, the performance may degrade if the simplicity bias is manifested. This test allows us to systematically investigate if the neural network utilizes the rest of the features f_i ∖ k or is predisposed to utilize feature f_k due to it being simpler than other features.




§ EXPERIMENTS




 §.§ Synthetic Data Engineering
In this section, we will describe the dataset generation for both the digit classification and the echocardiographic view classification problem. The classification network was trained using real data augmented by the synthetic data in varying proportions to study the impact of the synthetic simplicity. Let D^Tr_R,C_1 denote the set of all real images of class 1, D^Tr_S,C_1 denote the set of all synthetic images of class 1, D^Tr_R,C_2 denote the set of all real images of class 2, and D^Tr_S,C_2 denote the set of all synthetic images of class 2. The composition of the training data is thus defined as follows:


    |D^Tr| = α |D^Tr_R,C_1| + (1 - α) |D^Tr_S,C_1| + (1 - α) |D^Tr_R,C_2| + α |D^Tr_S,C_2|


where α∈ [0, 1] is a hyperparameter and α|D^Tr_R,C_1| + (1-α) |D^Tr_S,C_1| = (1-α)|D^Tr_R,C_2| + α|D^Tr_S,C_2|=k_Tr. The test set has an equal number of real and synthetic images for both classes, i.e., |D^Ts_R,C_1| = |D^Ts_S,C_1| = |D^Ts_R,C_2| = |D^Ts_S,C_2|=k_Ts.

We design the experiments to include a mix of real and synthetic images of class 1 and 2 in the training dataset D^Tr, controlled by the hyperparameter α.
When α=1, the training dataset D^Tr consists entirely of real images of class 1 (D^Tr_R,C_1) and synthetic images of class 2 (D^Tr_S,C_2). When α = 0, the training dataset D^Tr consists entirely of synthetic images of class 1 (D^Tr_S,C_1) and real images of class 2 (D^Tr_R,C_2).

For values of α between 0 and 1, the training dataset D^Tr is a mix of real and synthetic images of both classes 0 and 1, with α determining the specific proportion. If the model is able to maintain consistent performance regardless of the value of α, it can be considered robust. Otherwise, it may be affected by synthetic simplicity.



  §.§.§ Downstream Network
- In the case of digit classification task, we employed a simple Convolutional Neural Network (CNN) <cit.> as the downstream task. The model was trained on 6000 images each of digits 2 and 4 (k_Tr=6000) , using the Adam optimizer with a learning rate of 0.001, a batch size of 128, and the cross-entropy loss criterion, over the course of 50 epochs. Evaluation was conducted on a separate dataset comprising 1000 real and synthetic images for each digit 2 and 4 (k_Ts=1000).

Similarly, for the echocardiogram view classification task, we utilized ResNet-18 <cit.> as the downstream network. The model was trained on 400 views each of the two-chamber and four-chamber echocardiograms (k_Tr=400) using the Adam optimizer with a learning rate of 0.001, a batch size of 64, and the cross-entropy loss criterion. The training was conducted for 15 epochs. Evaluation was performed on a separate test set containing 100 real and synthetic views each for the two-chamber and four-chamber echocardiograms (k_Ts=100).




  §.§.§ Diffusion Model
- To generate synthetic digits, we employed the open-source generative diffusion  <cit.> model. And, in order to generate synthetic samples for echocardiograms, we utilized a pre-trained modified Semantic Diffusion Model (SDM) provided by  <cit.>. The synthetic samples were created based on the conditioning of semantic segmentation maps within the SDM framework.

We describe the two datasets used for experiments in the sections below: 



 §.§ MNIST Digit Classification

We utilized the publicly available "Modified National Institute of Standards and Technology" (MNIST) <cit.> handwritten digits dataset for our experiment. It consists of 70,000 28×28 black-and-white images of handwritten digits. For our binary classification experiment, we focused on the “2" vs./ “4" classes in the MNIST dataset. Figure <ref> depicts the real and synthetic samples of the digits 2 and 4. 





 §.§ CAMUS Echocardiogram View Classification




After investigating the issue of synthetic simplicity in the digit classification task, we demonstrate the phenomenon on a representative medical imaging task of cardiac view classification from echocardiographs. We utilized the publicly available echocardiography dataset "Cardiac Acquisitions for Multi-structure Ultrasound Segmentation" (CAMUS)  <cit.> for this experiment. The CAMUS dataset comprises four types of cardiac views from over 500 subjects: two-chamber end-systolic (2CH ES), two-chamber end-diastolic (2CH ED), four-chamber end-systolic (4CH ES), and four-chamber end-diastolic (4CH ED). We focused on the "4CH ED" and "2CH ED" classes in the CAMUS dataset for classification task. Figure <ref> shows real and synthetic samples of 4CH ED and 2CH ED echocardiogram views. 



§ RESULTS AND DISCUSSION






Experimental results for the CAMUS and MNIST classification tasks are presented in Figures <ref> and <ref>, respectively. The following observations can be noted:


	
  * Poor Performance in Extreme Cases of α: In the extreme case of α = 1 (high positive correlation), the training dataset consists solely of real 2CH images and synthetic 4CH images. Under this condition, the model performs well on real 2CH images (Fig. <ref>(a)) and synthetic 4CH images (Fig. <ref>(d)). However, due to the simplicity bias, the model's performance on synthetic 2CH images (Fig. <ref>(b)) and real 4CH images (Fig. <ref>(c)) is nearly zero. 
	Similarly, in the extreme case of α = 0 (high negative correlation), the model performs poorly on real 2CH images (Fig. <ref>(a)) and synthetic 4CH images (Fig. <ref>(d)). The MNIST results in Fig. <ref> show a similar trend.
	
	
  * Implication of Synthetic Simplicity in Medical vs./ General Domains: We can observe that the drop in test accuracy for CAMUS in all four cases is significantly steeper and more sudden compared to MNIST, where the drop is more gradual.  We hypothesize that this variation between echocardiographic classification vs./ digit classification is due to the fact that the complexity of the former task is arguably more complex, which biases the network to rely more heavily on the simpler feature - source of data. This indicates that the impact of synthetic simplicity has more severe implications in the medical context compared to general contexts. The complexity of the feature set in medical image data likely contributes to this pronounced effect. This underscores the urgent need to address simplicity bias in the medical domain, where the stakes are considerably higher due to the critical nature of the applications. Ensuring that models generalize well and do not overly rely on simplistic and spurious features is crucial in medical imaging to avoid potentially harmful misclassification. 
	
	
  * Case of Balanced Augmentation: When the correlation between the task labels and the data source is set to around 0.5, representing a balanced case, the test accuracy is quite consistent and is not significantly affected by the value of α. This can be observed in both the CAMUS (Fig. <ref>) and MNIST (Fig. <ref>) results. This observation suggests that in order to mitigate the simplicity bias  while using the synthetic data, we need to adopt a balanced approach when augmenting synthetic data with real data to achieve robust evaluation. A balanced approach ensures that the model does not overly rely on spurious features that are easy to learn but irrelevant to the actual task. By maintaining an intermediate level of correlation, the model is encouraged to utilize more complex and relevant features, leading to better generalization.
	
	
  * Persistence of Synthetic Simplicity Across Different Downstream Networks:
	In our study, we employed a basic CNN architecture for digit classification and a more intricate ResNet model for echocardiogram view classification as the downstream network. Despite ResNet's increased complexity, we consistently observed synthetic simplicity across both tasks. This phenomenon persisted irrespective of the network's architecture and depth. Even with its deeper layers, ResNet tended to exploit simplistic and spurious features inherent in synthetic data, leading to diminished performance during testing. These findings highlight the pervasive nature of synthetic simplicity in downstream tasks and emphasize the need for cautious integration of synthetic data for augmentation purposes.






§ CONCLUSION


We clearly demonstrate that downstream tasks can leverage the binary feature of the dataset source (real v/s synthetic) for tasks such as classification. Therefore, it is crucial for practitioners to use synthetic data carefully during model development. This caution helps prevent the inclusion of spurious features and the generation of artificially high accuracy values on a test set derived from the same distribution. We hope that this work will stimulate more interest in understanding the risks associated with using synthetic data without domain expertise and encourage further research.







 splncs04
 



 Paper Title: Navigating Text-to-Image Generative Bias across Indic Languages.





 










S. Mittal et al.




Department of CSE, IIT Jodhpur, Rajasthan, India Meta, Menlo Park, California, USA
 Weir P.B.C., Alameda, California, USA

Navigating Text-to-Image Generative Bias across Indic Languages
    Surbhi Mittal10000-0001-8910-4161 Arnav Sudan1 Mayank Vatsa10000-0001-5952-2274 Richa Singh10000-0003-4060-4573 

Tamar Glaser20009-0007-9322-2444 Tal Hassner30000-0003-2275-1406
    
======================================================================================================================================================================================



§ ABSTRACT 
This research investigates biases in text-to-image (TTI) models for the Indic languages widely spoken across India. It evaluates and compares the generative performance and cultural relevance of leading TTI models in these languages against their performance in English. Using the proposed IndicTTI benchmark, we comprehensively assess the performance of 30 Indic languages with two open-source diffusion models and two commercial generation APIs. The primary objective of this benchmark is to evaluate the support for Indic languages in these models and identify areas needing improvement. Given the linguistic diversity of 30 languages spoken by over 1.4 billion people, this benchmark aims to provide a detailed and insightful analysis of TTI models' effectiveness within the Indic linguistic landscape. The data and code for the IndicTTI benchmark can be accessed at <https://iab-rubric.org/resources/other-databases/indictti>.









§ INTRODUCTION



Text-to-image (TTI) generative technologies have transformed the landscape of digital media, distinguished by their extraordinary ability to produce detailed and visually engaging images from written text <cit.>. These models have found widespread application in a variety of fields, including gaming, animation, architecture, and fashion, offering significant benefits in terms of enhancing creativity and streamlining production timelines. However, despite the expansion of access through open-source models, challenges persist in achieving linguistic inclusivity <cit.>. The datasets these models are trained on are predominantly composed of English-language content, derived from both proprietary sources<cit.> and internet scraping <cit.>. This reliance on English-centric data adversely affects the quality of images generated from text in languages other than English.

As the integration of TTI models into various industries deepens, it becomes imperative to conduct a thorough evaluation of their performance and the diversity of the outputs they produce. While there has been research to study biases in TTI models<cit.>, the current evaluation landscape is notably lacking in benchmarks capable of effectively measuring these models' performance across a diverse linguistic spectrum, often overlooking the rich variety of global languages  <cit.>. To address this gap, we introduce the IndicTTI benchmark, an evaluation benchmark designed to explore and measure the biases present in text-to-image generation technologies, with an emphasis on a wider range of languages beyond those typically studied. This benchmark seeks to explore the linguistic diversity of a global population exceeding 1.4 billion, aiming not just to spotlight the shortcomings of existing TTI models but also to foster a broader application of these technologies that is truly inclusive and reflective of the myriad languages and cultures worldwide. Through this in-depth analysis, our goal is to spur improvements in TTI technologies, ensuring they cater to a wider audience and promote a more inclusive environment within digital media creation. 

In this research, we examine the performance of existing models in 30 languages other than English, with a specific focus on Indic languages, which are spoken by over a billion people worldwide. We believe that this benchmark is a step toward analyzing TTI models for a rich and diverse culture with a large population <cit.>. The 30 languages are written in 10 different scripts and belong to multiple language families. Among these, we have included the Scheduled Languages, which are considered the major literary languages of India and have a significant volume of literature <cit.>. 
We study two aspects of TTI models for assessing bias towards Indic languages, (i) model performance and (ii) model representation. We identify issues of correctness in generation, the presence of cultural elements, and the differences between open-source and API-based models. We utilize Stable Diffusion <cit.> and Alt Diffusion <cit.> models as part of our open-source analysis, and Midjourney <cit.> and Dalle3 <cit.> as part of the API-based generative models. We propose the Cyclic Language-Grounded Correctness (CLGC) metric to evaluate model performance and the Self-Consistency Across Languages (SCAL) metric to evaluate representation bias in the model generations. We further evaluate the model performance through Language-Grounded Correctness (LGC) and Image-Grounded Correctness (IGC) metrics.  Through our analysis, we observe how certain language scripts evoke imagery of Indian Gods in Midjourney, the overwhelmingly high generation of Asian women and couples in open-source models, and the high accuracy of generations in Dalle3 (Refer Fig. <ref>). Finally, we discuss the correlation between language script and cultural influences observed in the models.







§ RELATED WORK


Limited studies have been conducted to benchmark the performance of TTI models across different languages<cit.>. In NLP-based tasks, MEGA <cit.> benchmarks generative Large Language Models (LLMs) across 16 NLP datasets covering 70 diverse languages. This evaluation compares the performance of generative LLMs like Chat-GPT and GPT-4. Similarly, with a specific focus on the Indic languages, in IndicTrans2<cit.>, the authors provide a transformer-based multilingual NMT model that supports high-quality translations across all the 22 scheduled Indic languages. 


In the domain of text-to-image generative models, the recent Coco-crola benchmark <cit.> focuses on the evaluation of DallE and Stable Diffusion models for seven languages, namely English, Spanish, German, Chinese, Japanese, Hebrew, and Indonesian. Among these languages, English, German, and Indonesian share the Latin script, whereas Japanese and Chinese use the same script. Further, the authors focus on the generation of simple concepts through translation; however, in everyday applications, users are more likely to generate images through complex prompts. With this in mind, we focus on more complex prompts describing multiple elements in the generated image. The authors also propose the distinctiveness, coverage, and self-consistency measures.  Very limited research has also been conducted on the design of multilingual text-to-image generative models <cit.>. One such model is the  AltDiffusion model, which is a multilingual TTI diffusion model designed to process multiple languages <cit.>. Recently, Struppek et al. <cit.> have shown the influence of script-specific characters on the cultural aspects of the generated images.


In this work, we study the correctness and cultural representation bias of the TTI models, with a focus on Indic languages. However, the metrics and analysis in this study can be organically extended for the analysis of more languages.







§ BENCHMARK DESIGN

In this work, we propose the IndicTTI benchmark, which comprises of 31 languages, evaluated with four text-to-image generative models with six evaluation metrics covering aspects of correctness and representation. We describe the various design components of the benchmark below.



 §.§ Indic Languages and Prompts


For this benchmark, we utilize 30 Indic languages in addition to the English language. The languages are listed in Table <ref>. For prompts, we utilize the COCO-NLLB dataset <cit.>, which contains image-caption pairs for over 500K images, along with captions translated into 200 languages. In previous work<cit.>, the authors use simplistic concepts for prompts such as “a photo of a plane." However, the expectation from these models is to yield accurate and unbiased results in day-to-day usage. We believe that this expectation is better met with more complex prompts such as those present in the COCO-NLLB dataset. Therefore, we sample 1000 diverse image-caption pairs from the dataset. The dataset contains 24 Indic languages for which the captions are directly taken. These 24 languages do not include 4 languages that form a part of the 22 scheduled languages of India as per the Indian Constitution, namely Bodo, Dogri,  Konkani, and Santali. Using the IndicTrans2 translator <cit.>, we translate English captions to these languages along with two other languages, Manipuri (in Meitei script) and Sindhi (in Devanagari script). This leads to a total of 1000 prompts in 31 languages, including English, along with an associated image. 



 §.§ TTI Models and Generated Images

We utilize four different text-to-image models for the benchmark. For open-source models, we use the Stable Diffusion <cit.> and AltDiffusion Models<cit.>. For API-based models, we use Dalle3<cit.> and Midjourney<cit.>.  

Using the open-source models, we generate 4 images per prompt for the selected 1000 prompts across 31 languages, leading to 124K images corresponding to Stable Diffusion and AltDiffusion each. For API-based models, we generate over a subset of 200 prompts for Midjourney and over 50 prompts for Dalle3. This leads to a total of 24.8K images for Midjourney and 6.2K images for Dalle3. The evaluation is done over these sets and additionally over a subset of 40 prompts common across all models. The Dalle3 and Midjourney APIs are paid, and further details about the generation process are provided in the supplementary.



§ EVALUATION METHODS

In the IndicTTI benchmark, we focus on two aspects of TTI model evaluation- correctness and representation. Correctness refers to the ability to measure the semantic faithfulness with which the model generates the images for the corresponding prompt. On the other hand, representation refers to the measurement of the diversity of generated images within and across the different languages for the given prompts. The complete pipeline of the study is depicted in Fig. <ref>.



We adopt the following notation to describe the metrics and the evaluation pipeline. Let the set of languages be L. The set of prompts is denoted by P = {p_1, p_2,.. p_k}, and the corresponding representative images are Q = {q_1, q_2,.. q_k}. For each prompt in P, we generate n images, resulting in a total of n|P||L| images for a given generative model g. The set of images generated by g for a prompt p_k in language l ∈ L is denoted as,

    I_p_k, l = {i_p_k, l, r}_r=0^n

where, i_p_k, l, r denotes the image generated using g as g(p_k, l). Utilizing this notation, we define the metrics in this section. Across all the metrics, we compute high-level semantic features of images (using feature extractor f) and text (using feature extractor t) for effective comparison using the similarity function ϕ. 



 §.§ Correctness-based Metrics

We employ three correctness-based metrics in our study, including the proposed Cyclic Language-Grounded Correctness metric. The image-grounded and language-grounded correctness metrics are inspired by the correctness metric in CocoCrola<cit.>. These three metrics comprehensively evaluate the generated images across text-text, image-image, and image-text spaces. A higher correctness value indicates greater accuracy of the generative model in producing images that are faithful to the text.

Cyclic Language-Grounded Correctness (CLGC): This metric evaluates the generated images I in the text-text space. This is achieved in a cyclic manner through generating captions over I using a caption generator c, as,

    p'_k = c(i_p_k, l, r)

where, p'_k denotes the generated caption for r_th image in set I_p_k,l. Then, with p_k and p'_k as prompts in the English language for the language l, we obtain,

    CLGC(l) = 1/n|P|∑_p_k^P∑_r=0^nϕ [  t(p_k),  t(p'_k)  ]



Image-Grounded Correctness (IGC): For this metric, the correctness of the generated images is evaluated in the image-image space, as,

    IGC(l) = 1/n|P|∑_p_k^P∑_r=0^nϕ [  f(q_k),  f(i_p_k)  ]

where, the ground-truth image q_k is compared to each of the n generated images for the prompt p_k.

Language-Grounded Correctness (LGC): This metric evaluates correctness in the image-text space with p_k denoting the prompt in the English language,

    LGC(l) = 1/n|P|∑_p_k^P∑_r=0^nϕ [  t(p_k),  f(i_p_k)  ]





 §.§ Representation-based Metrics

In this section, we discuss the three metrics utilized for measuring the representativeness of concepts across the languages. We propose the Self-Consistency Across Languages metric to capture the bias in representation across the different languages. The other two metrics are inspired by the self-consistency and distinctiveness metrics in previous work<cit.>.

Self-Consistency Across Languages (SCAL): This metric captures the variation in the generated images for the same prompt across the different languages. So, for a language pair (l_a, l_b),

    SCAL(l_a, l_b, p_k)    = 1/n2∑_u=0^n∑_v=0^nϕ [  f(i_p_k, l_a, u),  f(i_p_k, l_b, v)  ]; u  v 
    
        SCAL(l_a, l_b)    = 1/|P|∑_p_k^P SCAL(l_a, l_b, p_k)

Then, the overall SCAL score for the given set of languages becomes,

    SCAL = 1/|L|2∑_l_a^L∑_l_b^L SCAL(l_a, l_b);  a  b

The lower the value of SCAL, the less consistent the generations are across the different languages. On the other hand, a higher value of SCAL would demonstrate high consistency between concepts across the different languages.

Self-Consistency Within Language (SCWL): This metric is computed within the images generated for a particular prompt for a given language, as,

    SCWL(l, p_k)    = 1/n2∑_u=0^n∑_v=0^nϕ [  f(i_p_k, u),  f(i_p_k, v)  ]; u  v 
    
        SCWL(l)    = 1/|P|∑_p_k^P SCWL(l, p_k)

A high self-consistency showcases good consistency between the semantic content of the generated images.

Distinctiveness Within Language (DWL): This metric depicts the diversity across the images generated for the different prompts p_a and p_b,

    SWL(l)    = 1 - 1/|P|2∑_p_a^P∑_p_b^Pϕ [  f(i_p_a),  f(i_p_b)  ];  a  b


where a higher distinctiveness score showcases the capability of the model to generate diverse images with varying prompts.

In addition to quantitative evaluation, we conduct an extensive qualitative evaluation of the generated images with respect to the captions, languages, and scripts. The implementation details are provided in the supplementary.




Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in the IndicTTI benchmark. Existing models provide high correctness for English languages while providing lower values for Indic languages.  


Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     64.53     7.29                                                      6.39                                                     7.24                                                     7.35                                                     7.13                                                     6.91                                                     7.53                                                     6.50                                                     8.03                                                     8.36                                                     8.78                                                      8.14                                                     11.29                                                    7.30                                                     7.65                                                 

Alt Diffusion        58.16     14.65                                                     16.70                                                    15.97                                                    16.79                                                    22.98                                                    16.08                                                    16.50                                                    12.17                                                    17.97                                                    18.82                                                    17.64                                                     17.99                                                    14.82                                                    18.10                                                    14.50                                                

Midjourney           64.31     12.99                                                     11.89                                                    12.49                                                    11.31                                                    12.12                                                    10.92                                                    11.99                                                    9.80                                                     11.89                                                    12.56                                                    9.32                                                      12.19                                                    13.49                                                    12.84                                                    12.16                                                

Dalle3               65.70     53.36                                                     60.73                                                    57.25                                                    64.16                                                    60.13                                                    57.68                                                    61.84                                                    35.24                                                    59.90                                                    53.96                                                    62.67                                                     51.22                                                    45.19                                                    53.98                                                    55.26                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     64.53     10.08                                                     6.29                                                     7.52                                                     7.13                                                     10.11                                                    7.12                                                     8.25                                                     7.66                                                     6.25                                                     6.26                                                     7.95                                                      7.78                                                     10.51                                                    7.59                                                     10.17                                                

Alt Diffusion        58.16     17.68                                                     15.96                                                    15.46                                                    16.93                                                    16.32                                                    15.41                                                    16.74                                                    16.80                                                    13.04                                                    11.98                                                    13.50                                                     15.82                                                    14.13                                                    13.37                                                    15.25                                                

Midjourney           64.31     15.38                                                     12.72                                                    12.37                                                    11.92                                                    13.04                                                    11.47                                                    12.06                                                    12.53                                                    11.96                                                    12.34                                                    11.77                                                     12.59                                                    11.36                                                    12.35                                                    9.72                                                 

Dalle3               65.70     64.02                                                     47.45                                                    55.51                                                    58.40                                                    57.37                                                    60.75                                                    59.18                                                    60.42                                                    49.03                                                    28.87                                                    59.84                                                     50.18                                                    10.93                                                    56.78                                                    11.19                                                
 




Image-Grounded Correctness (IGC) (%) across the different Indic languages in the IndicTTI benchmark. 

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     51.21     24.19                                                     23.84                                                    23.25                                                    23.53                                                    23.24                                                    22.96                                                    23.53                                                    23.58                                                    23.96                                                    23.76                                                    22.97                                                     23.11                                                    23.06                                                    22.68                                                    22.61                                                

Alt Diffusion        46.58     24.23                                                     25.08                                                    25.18                                                    25.43                                                    27.74                                                    24.27                                                    25.42                                                    22.79                                                    25.98                                                    26.27                                                    25.48                                                     25.68                                                    24.68                                                    25.06                                                    22.19                                                

Midjourney           48.94     22.69                                                     22.66                                                    23.06                                                    22.50                                                    22.81                                                    22.04                                                    22.75                                                    21.97                                                    22.29                                                    22.82                                                    23.05                                                     22.81                                                    23.34                                                    22.55                                                    22.09                                                

Dalle3               48.92     40.02                                                     44.22                                                    43.75                                                    44.86                                                    42.74                                                    43.12                                                    46.10                                                    30.91                                                    43.14                                                    40.00                                                    45.06                                                     37.66                                                    36.30                                                    41.25                                                    41.39                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     51.21     22.92                                                     22.58                                                    23.41                                                    23.71                                                    25.16                                                    22.98                                                    23.19                                                    23.41                                                    24.00                                                    23.83                                                    22.90                                                     23.88                                                    24.14                                                    23.64                                                    23.98                                                

Alt Diffusion        46.58     25.40                                                     24.45                                                    25.01                                                    26.39                                                    25.97                                                    25.53                                                    25.59                                                    26.10                                                    24.60                                                    23.82                                                    24.87                                                     24.73                                                    24.16                                                    25.16                                                    24.35                                                

Midjourney           48.94     23.30                                                     22.76                                                    22.63                                                    22.90                                                    23.09                                                    22.65                                                    22.97                                                    22.64                                                    22.16                                                    22.57                                                    22.90                                                     22.79                                                    21.25                                                    22.53                                                    21.99                                                

Dalle3               48.92     44.10                                                     38.81                                                    44.04                                                    43.18                                                    44.88                                                    44.22                                                    44.03                                                    44.17                                                    37.46                                                    29.76                                                    44.69                                                     38.91                                                    23.18                                                    42.18                                                    22.46                                                
 




Language-Grounded Correctness (LGC) (%) across the different Indic languages in the IndicTTI benchmark.  

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     33.98     3.90                                                      3.26                                                     3.28                                                     3.79                                                     4.21                                                     4.09                                                     3.61                                                     3.50                                                     3.68                                                     3.56                                                     3.01                                                      3.77                                                     4.16                                                     4.23                                                     4.09                                                 

Alt Diffusion        30.90     6.50                                                      6.87                                                     7.34                                                     7.56                                                     10.67                                                    6.53                                                     7.81                                                     4.54                                                     8.25                                                     8.57                                                     7.87                                                      8.49                                                     6.53                                                     7.12                                                     6.00                                                 

Midjourney           32.25     2.08                                                      1.95                                                     2.86                                                     3.27                                                     4.03                                                     2.44                                                     2.87                                                     1.73                                                     3.36                                                     3.00                                                     2.59                                                      3.41                                                     3.91                                                     3.92                                                     2.76                                                 

Dalle3               33.06     26.42                                                     30.54                                                    28.04                                                    31.57                                                    28.48                                                    28.22                                                    30.87                                                    14.98                                                    29.17                                                    26.67                                                    31.41                                                     24.32                                                    19.92                                                    26.24                                                    26.68                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     33.98     5.85                                                      3.80                                                     4.12                                                     4.26                                                     5.83                                                     4.10                                                     4.25                                                     4.45                                                     3.50                                                     3.90                                                     3.55                                                      4.11                                                     3.71                                                     4.11                                                     4.52                                                 

Alt Diffusion        30.90     7.72                                                      5.41                                                     7.22                                                     8.22                                                     8.04                                                     7.82                                                     7.70                                                     7.81                                                     5.99                                                     4.73                                                     5.98                                                      6.12                                                     5.76                                                     6.82                                                     6.30                                                 

Midjourney           32.25     3.99                                                      3.19                                                     3.33                                                     3.66                                                     4.30                                                     2.95                                                     4.35                                                     3.31                                                     1.78                                                     3.68                                                     3.56                                                      2.86                                                     2.45                                                     3.32                                                     2.64                                                 

Dalle3               33.06     30.59                                                     23.09                                                    28.30                                                    28.68                                                    28.91                                                    29.99                                                    30.02                                                    29.39                                                    22.86                                                    12.93                                                    29.10                                                     24.05                                                    2.09                                                     27.70                                                    1.52                                                 
 









§ BENCHMARK RESULTS AND ANALYSIS

In this section, we report results on the 4 models used for evaluation, namely Stable Diffusion, AltDiffusion, Midjourney, and Dalle3, on a common set of prompts.  We provide the results on the complete set of prompts for the models in the supplementary. The observations are consistent with those made on the common subset reported in this section.



Correctness Bias: As discussed in the previous section, we evaluate the correctness of generation across the Indic languages through the metrics of CLGC, IGC, and LGC, which correspond to the text-text, image-image, and image-text spaces, respectively. In the text-text space, we observe that the CLGC metric is significantly high for the English language (Refer Table <ref>).  This demonstrates that the generated images have high faithfulness to the text, as the captions generated for the generated images are semantically closest to the original English caption. All models have a high CLGC value of >60% for the English language. On the other hand, CLGC values drop significantly for Indic languages for Stable Diffusion and AltDiffusion models, with the performance dropping to 7.29% from 64.53% and 14.65% from 58.16%, respectively for the Assamese language. The values of CLGC tend to be lower for languages with a Bengali script, namely Manipuri (Beng), Assamese, and Bengali. On computing the average across the 30 Indic languages and comparing the CLGC scores with those obtained for the English language, there is a 56.64% drop in performance for the Stable Diffusion model. Similarly, there is a 42.16%, 52.16%, and 13.62% drop for AltDiffusion, Midjourney, and Dalle3, with Dalle3 providing the best performance.

Similarly, in the image-image space, we observe higher IGC metric values for the English language, showcasing up to 50% image semantic similarity with the ground truth images. With Indic languages, there is an average drop of 27.74% and 21.53% for the Stable Diffusion and AltDiffusion models. While there is a high drop of 26.32% for Midjourney generated images, the drop is significantly lesser for Dalle3 at 8.70%, signifying a superior understanding of Indic languages. For image-text similarity, we use the English language as the prompt of comparison. We observe higher LGC metric values for the images generated using English prompts, in the range of 30-35% across the models. These values drop sharply for Stable Diffusion, AltDiffusion, and Midjourney, with average LGC values of 4.01%, 7.08%, and 3.12% across the Indic languages, respectively. Dalle3 has the highest LGC value at 25.09%, with extremely low values observed for the Manipuri (Meitei script) (1.52%), Santali (2.09%), and Bodo (12.93%) languages.

Across all the metrics, we observe that Dalle3 outperforms all other models when evaluated for Indic languages with a significant margin. However, Dalle3 struggles with the Manipuri (Meitei script), Santali, and Bodo languages. Other models struggle to generalize on the Indic languages with AltDiffusion performing the best among the other three models. This behavior could be a result of increased generalizability due to the multilingual training of the model. On the other hand, while all models perform similarly for the English language, AltDiffusion performs the worst, possibly due to catastrophic forgetting when trained for multilinguality.  






Representativeness Bias: To better understand the representative capacity of the models, we evaluate the SCAL, SCWL, and DWL metrics as explained in the previous section. While these metrics do not contain the capability to estimate the correctness of generation on their own, their values showcase important behavior in terms of diversity in model generation. 

The SCAL metric evaluates the self-consistency of generated images across the different prompts. A high SCAL value represents the model's ability to consistently generate similar semantic content across languages. In Fig. <ref>, we plot the SCAL values across the different language pairs for the four models. From Fig. <ref>(a), we observe that there is little semantic similarity of concepts between the English language and other languages. Since we know that these models are primarily trained to perform well on English, this is also an indicator of poor performance across the other languages. Interestingly, the concepts generated across other languages have higher overlap showing that they generate images that are far similar in semantic content, despite being possibly incorrect in their generations. In the previous section, we observed that the performance of generation for Indic languages is only high in Dalle3. This is demonstrated well in Fig. <ref>(d) where we observe that the semantic similarity between English and Indic languages is high, with performance being poorest between Santali and Manipuri (Meitei script), followed by Bodo and Manipuri (Bengali script). 


The overall SCAL metric value as per Eqn. <ref> for the Stable Diffusion, AltDiffusion, Midjourney, and Dalle3 models is 25.44%, 23.73%, 26.75%, and 29.90%, respectively. This indicates an overall high consistency of Dalle3 in generating concepts across different languages. It is also evident from Fig. <ref> that the Stable Diffusion, AltDiffusion, and Midjourney generators suffer greatly in maintaining semantic consistency with images generated via English, whereas they tend to generate similar content for other languages. From Fig. <ref>(c), we also observe that Midjourney has high semantic consistency between images generated from languages with the Devanagari script, highlighting the language script plays a crucial part in the semantic content of the image.



Next, we report the representativeness of different prompts within the same language through the SCWL metric. In Fig. <ref>(a), we observe that the self-consistency of models is higher for the English language. A higher value demonstrates a more stable representation of concepts in the model. For Dalle3, the SCWL values for the different languages are comparable to those of English but still lower. In agreement with previous observations, the values are especially poor for the Santali, Manipuri (Meitei script), Bodo, and Manipuri (Bengali script) languages. In Midjourney, the values stay consistent, with the exception of the Santali and Manipuri (Meitei) languages. Similar trends are observed for the other models. When compared with other metrics, there is a low variation in the values of self-consistency within the language for the different models and languages. This behavior could be the result of the relative stability of the model in its generation. While the model may generate incorrectly, it associates a given text for generation in the latent space consistently. Therefore, the model may demonstrate high self-consistency by generating images that are related to a given prompt, albeit incorrectly. 

Finally, in Fig. <ref>(b), results are presented for the DWL metric. The DWL metric showcases how distinct are the generations corresponding to the different prompts. This metric is dependent on the relatedness of the prompts used for generation. While sampling prompts for the IndicTTI benchmark, we ensured semantic diversity in the selected 1000 prompts. The details of the selection process are provided in the supplementary. It is evident from Fig. <ref>(b) that Dalle3 provides the highest level of distinctiveness between the generated prompts across the different languages, only suffering for the Santali and Manipuri (Meitei script) languages. Surprisingly, Midjourney has the lowest distinctiveness in its generations across the Indic languages, indicating it generates similar content over the different prompts. Similarly, the Stable Diffusion model performs poorly. The AltDiffusion model provides the second highest distinctiveness among the four models. This could be attributed to its multilingual training, making it sensitive to different scripts during generation.    








§ QUALITATIVE ANALYSIS

In this section, we qualitatively analyze the images generated by the different models across the different languages. Based on our observations in the previous section, we explore the generated images and study the alignment between the quantified metrics and generated images.

As in the previous section, we study two aspects of bias, correctness-based as well as representation-based. In Fig. <ref>, we observe that the Stable Diffusion and AltDiffusion models suffer greatly when prompted with Indic languages. The images generated using the prompts in Indic languages often do not resemble the description of the actual prompts. For example, when prompted to generate a swan in the Punjabi language, the Stable Diffusion model generates an image of 3 men in turbans, whereas the AltDiffusion model generates the image of a couple. Here again, the man is observed to be wearing a turban. We further observe a relatively high depiction of people in images, specifically those of Indian ethnicity, even when the actual content of the prompt does not involve any mention of people. Similarly, the lack of correspondence between the actual prompt and the generated images for Midjourney can be seen in Fig. <ref>. For example, on prompting Midjourney to generate images of “a man in a brown jacket," it generates stereotypically Indian women. Other simplistic prompts in the Hindi language also lead to incorrect generations. 



For Dalle3, we observe that the model generates correct images, relevant to the content of the prompt. This observation is corroborated by the high performance of Dalle3 on the metrics proposed with the benchmark. However, the Dalle3 model struggles with certain languages, such as Manipuri, Santali, and Bodo. In order to better understand the poor quality of generations, we prompted the ChatGPT model, which uses Dalle3 for image generation, to generate some images for these languages. On providing the ChatGPT model with prompts in these languages, it seemed that the model did not understand these scripts at all. Specifically, it identified the script/language it was being prompted with and then continued to “imagine" an image based on cultural influences. Some of these examples are shown in Fig. <ref> where the model generates images influenced by the Manipuri, Nepali, and Santal cultures.

On observing the images for the different models (Refer Fig. <ref>), we notice a correlation between different language scripts and the images generated corresponding to them. All models, except Dalle3, frequently generate images of females and Indian religious figures such as Gods and pandits. Dalle3, on the other hand, tends to incorporate common Indian religious sites like temples and ghats. In Hindu culture, many women regularly wear sarees and bindi, and these elements are prominently featured in the images produced by both Stable Diffusion and AltDiffusion. 
Additionally, the Arabic script is frequently associated with elements of Muslim culture, such as the burqa and niqab. Elements of South Indian culture are often present in the images generated for prompts in Malayalam, a language widely spoken in Southern India. These correlations between script and generated images are common, especially in those produced by Midjourney. When prompted in Sanskrit, the sacred language of Hinduism and predominant in religious texts, the Stable Diffusion model is observed to generate images of Indian Gods. Similarly, prompts in Punjabi often result in images of men wearing turbans, reflecting the cultural and religious practices of Punjabi speakers. More observations are detailed in the supplementary materials.






§ CONCLUSION

In this research, we introduce the IndicTTI benchmark, a comprehensive framework that integrates a variety of prompts, several text-to-image (TTI) models, and an extensive assessment of generative biases using the proposed evaluation methods. Employing both performance-oriented and representativeness-centric metrics, we are able to thoroughly assess the capacity of the models for multilingual generation. The qualitative analysis further emphasizes the extent of cultural influences embedded within the models' outputs. While the presence of cultural elements is not inherently detrimental, some generated images may inadvertently reinforce cultural stereotypes. The IndicTTI benchmark makes a dual contribution: firstly, it establishes a robust quantitative standard for assessing the multilingual capabilities of TTI models; secondly, it highlights the incorporation of Indian cultural motifs in the imagery produced.

Limitations:
In this paper, we conduct both quantitative and qualitative evaluations of various aspects of TTI systems in a multilingual context. The prompts used in our benchmark are primarily derived from COCO-NLLB, which translates English captions from the COCO dataset into multiple languages, or translated from English using the open-source IndicTrans2 model. Therefore, the generation performance for Indic prompts is indirectly affected by the quality of these translations. A user study, detailed in the supplementary materials, briefly examines the translation quality and demonstrates the appropriateness and accuracy of the translated captions. While the high performance of the Dalle3 model suggests that the translations are meaningful, it is essential to recognize the influence of translation in the benchmarking process. Furthermore, the semantic similarity between images and text is evaluated using large pre-trained models, which may possess inherent biases that could inadvertently impact the evaluation. Our comprehensive qualitative assessment generally supports the quantitative results derived from these pre-trained models. Nonetheless, there remains a possibility that inadvertent biases could affect the outcomes.



§ ACKNOWLEDGEMENTS

S. Mittal is partially supported by the UGC-Net JRF Fellowship and IBM fellowship. M. Vatsa was partially supported through the Swarnajayanti Fellowship. This work was partially supported by Facebook AI. All data was stored, and experiments were performed on IITJ servers by IITJ faculty and students.


splncs04









§ BENCHMARK DESIGN

In this section, we provide additional details about the benchmark design.



 §.§ Indic Languages and Prompts


In this research, we introduce the IndicTTI benchmark where we study the performance of popular text-to-image (TTI) models in 30 languages. While the benchmark may be extended for any number of languages, we select 30 Indic languages, which are written in 10 different scripts and have roots in multiple language families. Detailed information about the different languages is provided in Table <ref>.



For prompts, we utilize the COCO-NLLB dataset <cit.>, which contains image-caption pairs for over 500K images, along with captions translated into 200 languages. We sample 1000 diverse image-caption pairs from the dataset. In order to avoid prompts with proper nouns, such as names of celebrities and/or brand names, we filtered the dataset to remove any captions that contained capitalized words in the middle of the sentence. From the filtered dataset, we randomly subsample 1000 captions. Next, for a diverse selection, we computed sentence-level embeddings for the 1000 prompts using a SentenceFormer model and calculated the average similarity between any two prompts. This experiment was repeated for 1000 iterations, and the subset with the lowest sentence similarity between prompts was selected. The subsets of 200 and 50 prompts for Midjourney and Dalle3, respectively, were chosen randomly from the selected subset of 1000 prompts. 



 §.§ TTI Models and Generated Images

We utilize four different text-to-image models for the benchmark. For open-source models, we use the Stable Diffusion and AltDiffusion Models (without safety filters). In the main paper, we report results on the m2 variant of AltDiffusion trained on the English and Chinese languages. Extended results, including the m9 variant trained on the English, Chinese, Spanish, French, Russian, Japanese, Korean, Arabic, and Italian languages, are reported in the supplementary. The extended results exhibit a similar pattern to the m2 variant. Detailed results are provided in the subsequent sections. Unless specified otherwise, the AltDiffusion model refers to the m2 variant in this work. For API-based models, we use Dalle3<cit.> and Midjourney<cit.>, which are both paid. Stable Diffusion and AltDiffusion models generate images of size 512 x 512, whereas Midjourney and Dalle3 generate images of size 1024 x 1024.






 §.§ Quality of Translated Captions

Our prompts were primarily sourced from COCO-NLLB or translated from English using the IndicTrans2 model. As specified in the Limitations section of the main paper, this suggests that the translation quality may have influenced the performance of image generation with Indic prompts. To evaluate the translation quality, we conducted a user study on a common set of 40 prompts across 30 languages, validated by 65 annotators with an average proficiency of 4.78 out of 5 in at least one Indic language. We followed the XSTS protocol <cit.>, also used to assess the translation quality of the NLLB model <cit.>. On a scale from 1 (worst) to 5 (best), the average translation quality for all languages is reported in Fig. <ref>, indicating that the sentences are mostly equivalent or paraphrases of each other, according to the XSTS protocol, and thus suitable for image generation. In this experiment, the inter-rater agreement, measured through percent agreement, was 74.8%. For languages such as kas_Deva, snd_Deva, and sin_Sinh, the translation quality of certain prompts was observed to be less than 3 on the XSTS scale.



§ IMPLEMENTATION DETAILS

In this section, we discuss the implementation details involved in the creation of the IndicTTI benchmark as well as its evaluation.

For generation using open-source TTI systems of Stable Diffusion[<https://huggingface.co/runwayml/stable-diffusion-v1-5>] and AltDiffusion[<https://huggingface.co/BAAI/AltDiffusion>, <https://huggingface.co/BAAI/AltDiffusion-m9>], we utilized the models available at HuggingFace. Dalle3 API was accessed through a Python script whereas the generation for Midjourney was done using Discord. For Stable Diffusion and AltDiffusion, the generation was done using an NVIDIA DGX Station consisting of 4 NVIDIA V100 GPU with 32 GB VRAM each, using a batch size of 8. The generation was repeated using 4 seeds, providing 4 images for every prompt. The DDIM Scheduler was used for inferencing with 50 steps.

For evaluation, all experiments are conducted on LINUX-based systems using Python-based libraries, and specifically, the PyTorch library is used. To extract rich semantic text and image features for the evaluation metrics, we utilize various modules of the BLIP-2 model[<https://huggingface.co/Salesforce/blip2-opt-2.7b>]. We utilize the image encoder of the BLIP-2 model as the image feature extractor f for computing the CLGC, IGC, SCAL, SCWL, and DWL metrics. Additionally, for the CLGC metric, we utilize the image-captioning capabilities of BLIP-2 captioner c to generate captions for generated images, and for extracting rich textual features, the SentenceFormer model[<https://huggingface.co/sentence-transformers/all-mpnet-base-v2>]. For the LGC metric, we require image-text features that are extracted from BLIP-2 using the LAVIS[<https://github.com/salesforce/LAVIS/>] library. The similarity function ϕ is computed using the cosine similarity. The code for generation, as well as evaluation, can be 
accessed through <https://iab-rubric.org/resources/other-databases/indictti>.



Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in IndicTTI on the common set of prompts.  


Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     64.53     7.29                                                      6.39                                                     7.24                                                     7.35                                                     7.13                                                     6.91                                                     7.53                                                     6.50                                                     8.03                                                     8.36                                                     8.78                                                      8.14                                                     11.29                                                    7.30                                                     7.65                                                 

Alt Diffusion        58.16     14.65                                                     16.70                                                    15.97                                                    16.79                                                    22.98                                                    16.08                                                    16.50                                                    12.17                                                    17.97                                                    18.82                                                    17.64                                                     17.99                                                    14.82                                                    18.10                                                    14.50                                                

Alt Diffusion m9     53.19     18.02                                                     27.08                                                    26.50                                                    29.78                                                    26.66                                                    27.04                                                    28.21                                                    12.39                                                    31.42                                                    27.70                                                    27.14                                                     21.50                                                    18.97                                                    27.81                                                    24.34                                                

Midjourney           64.31     12.99                                                     11.89                                                    12.49                                                    11.31                                                    12.12                                                    10.92                                                    11.99                                                    9.80                                                     11.89                                                    12.56                                                    9.32                                                      12.19                                                    13.49                                                    12.84                                                    12.16                                                

Dalle3               65.70     53.36                                                     60.73                                                    57.25                                                    64.16                                                    60.13                                                    57.68                                                    61.84                                                    35.24                                                    59.90                                                    53.96                                                    62.67                                                     51.22                                                    45.19                                                    53.98                                                    55.26                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     64.53     10.08                                                     6.29                                                     7.52                                                     7.13                                                     10.11                                                    7.12                                                     8.25                                                     7.66                                                     6.25                                                     6.26                                                     7.95                                                      7.78                                                     10.51                                                    7.59                                                     10.17                                                

Alt Diffusion        58.16     17.68                                                     15.96                                                    15.46                                                    16.93                                                    16.32                                                    15.41                                                    16.74                                                    16.80                                                    13.04                                                    11.98                                                    13.50                                                     15.82                                                    14.13                                                    13.37                                                    15.25                                                

Alt Diffusion m9     53.19     26.29                                                     17.33                                                    26.03                                                    23.69                                                    28.06                                                    24.48                                                    29.98                                                    28.99                                                    26.24                                                    12.29                                                    21.57                                                     20.89                                                    12.97                                                    23.71                                                    13.36                                                

Midjourney           64.31     15.38                                                     12.72                                                    12.37                                                    11.92                                                    13.04                                                    11.47                                                    12.06                                                    12.53                                                    11.96                                                    12.34                                                    11.77                                                     12.59                                                    11.36                                                    12.35                                                    9.72                                                 

Dalle3               65.70     64.02                                                     47.45                                                    55.51                                                    58.40                                                    57.37                                                    60.75                                                    59.18                                                    60.42                                                    49.03                                                    28.87                                                    59.84                                                     50.18                                                    10.93                                                    56.78                                                    11.19                                                
 


Image-Grounded Correctness (IGC) (%) across the different Indic languages in IndicTTI on the common set of prompts. 

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     51.21     24.19                                                     23.84                                                    23.25                                                    23.53                                                    23.24                                                    22.96                                                    23.53                                                    23.58                                                    23.96                                                    23.76                                                    22.97                                                     23.11                                                    23.06                                                    22.68                                                    22.61                                                

Alt Diffusion        46.58     24.23                                                     25.08                                                    25.18                                                    25.43                                                    27.74                                                    24.27                                                    25.42                                                    22.79                                                    25.98                                                    26.27                                                    25.48                                                     25.68                                                    24.68                                                    25.06                                                    22.19                                                

Alt Diffusion m9     42.43     26.50                                                     28.39                                                    30.46                                                    28.94                                                    29.74                                                    29.59                                                    28.86                                                    23.29                                                    30.67                                                    29.55                                                    28.76                                                     26.54                                                    25.54                                                    28.29                                                    28.62                                                

Midjourney           48.94     22.69                                                     22.66                                                    23.06                                                    22.50                                                    22.81                                                    22.04                                                    22.75                                                    21.97                                                    22.29                                                    22.82                                                    23.05                                                     22.81                                                    23.34                                                    22.55                                                    22.09                                                

Dalle3               48.92     40.02                                                     44.22                                                    43.75                                                    44.86                                                    42.74                                                    43.12                                                    46.10                                                    30.91                                                    43.14                                                    40.00                                                    45.06                                                     37.66                                                    36.30                                                    41.25                                                    41.39                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     51.21     22.92                                                     22.58                                                    23.41                                                    23.71                                                    25.16                                                    22.98                                                    23.19                                                    23.41                                                    24.00                                                    23.83                                                    22.90                                                     23.88                                                    24.14                                                    23.64                                                    23.98                                                

Alt Diffusion        46.58     25.40                                                     24.45                                                    25.01                                                    26.39                                                    25.97                                                    25.53                                                    25.59                                                    26.10                                                    24.60                                                    23.82                                                    24.87                                                     24.73                                                    24.16                                                    25.16                                                    24.35                                                

Alt Diffusion m9     42.43     28.39                                                     25.05                                                    27.48                                                    27.34                                                    29.31                                                    28.24                                                    30.17                                                    29.53                                                    29.79                                                    23.28                                                    27.16                                                     25.83                                                    24.10                                                    28.24                                                    24.26                                                

Midjourney           48.94     23.30                                                     22.76                                                    22.63                                                    22.90                                                    23.09                                                    22.65                                                    22.97                                                    22.64                                                    22.16                                                    22.57                                                    22.90                                                     22.79                                                    21.25                                                    22.53                                                    21.99                                                

Dalle3               48.92     44.10                                                     38.81                                                    44.04                                                    43.18                                                    44.88                                                    44.22                                                    44.03                                                    44.17                                                    37.46                                                    29.76                                                    44.69                                                     38.91                                                    23.18                                                    42.18                                                    22.46                                                
 


Language-Grounded Correctness (LGC) (%) across the different Indic languages in IndicTTI on the common set of prompts.  

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     33.98     3.90                                                      3.26                                                     3.28                                                     3.79                                                     4.21                                                     4.09                                                     3.61                                                     3.50                                                     3.68                                                     3.56                                                     3.01                                                      3.77                                                     4.16                                                     4.23                                                     4.09                                                 

Alt Diffusion        30.90     6.50                                                      6.87                                                     7.34                                                     7.56                                                     10.67                                                    6.53                                                     7.81                                                     4.54                                                     8.25                                                     8.57                                                     7.87                                                      8.49                                                     6.53                                                     7.12                                                     6.00                                                 

Alt Diffusion m9     27.56     7.28                                                      10.92                                                    12.26                                                    12.58                                                    12.36                                                    11.58                                                    11.37                                                    3.66                                                     14.01                                                    11.73                                                    10.20                                                     9.93                                                     7.61                                                     11.39                                                    11.30                                                

Midjourney           32.25     2.08                                                      1.95                                                     2.86                                                     3.27                                                     4.03                                                     2.44                                                     2.87                                                     1.73                                                     3.36                                                     3.00                                                     2.59                                                      3.41                                                     3.91                                                     3.92                                                     2.76                                                 

Dalle3               33.06     26.42                                                     30.54                                                    28.04                                                    31.57                                                    28.48                                                    28.22                                                    30.87                                                    14.98                                                    29.17                                                    26.67                                                    31.41                                                     24.32                                                    19.92                                                    26.24                                                    26.68                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     33.98     5.85                                                      3.80                                                     4.12                                                     4.26                                                     5.83                                                     4.10                                                     4.25                                                     4.45                                                     3.50                                                     3.90                                                     3.55                                                      4.11                                                     3.71                                                     4.11                                                     4.52                                                 

Alt Diffusion        30.90     7.72                                                      5.41                                                     7.22                                                     8.22                                                     8.04                                                     7.82                                                     7.70                                                     7.81                                                     5.99                                                     4.73                                                     5.98                                                      6.12                                                     5.76                                                     6.82                                                     6.30                                                 

Alt Diffusion m9     27.56     10.38                                                     6.25                                                     10.74                                                    10.44                                                    12.38                                                    10.67                                                    12.58                                                    12.56                                                    11.04                                                    4.71                                                     9.69                                                      8.20                                                     5.16                                                     9.91                                                     5.65                                                 

Midjourney           32.25     3.99                                                      3.19                                                     3.33                                                     3.66                                                     4.30                                                     2.95                                                     4.35                                                     3.31                                                     1.78                                                     3.68                                                     3.56                                                      2.86                                                     2.45                                                     3.32                                                     2.64                                                 

Dalle3               33.06     30.59                                                     23.09                                                    28.30                                                    28.68                                                    28.91                                                    29.99                                                    30.02                                                    29.39                                                    22.86                                                    12.93                                                    29.10                                                     24.05                                                    2.09                                                     27.70                                                    1.52                                                 
 





Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in the IndicTTI benchmark on the complete set of prompts. Existing models provide high correctness for English languages while providing lower values for Indic languages. 


Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     67.97     7.56                                                      8.22                                                     8.08                                                     8.64                                                     8.16                                                     7.31                                                     8.15                                                     7.93                                                     9.09                                                     8.64                                                     8.60                                                      8.84                                                     12.22                                                    8.63                                                     9.36                                                 

Alt Diffusion        62.40     14.32                                                     16.87                                                    15.42                                                    17.34                                                    22.33                                                    15.67                                                    16.36                                                    12.07                                                    17.27                                                    18.78                                                    16.35                                                     16.54                                                    17.13                                                    17.69                                                    15.51                                                

Alt Diffusion m9     55.99     16.87                                                     27.08                                                    25.45                                                    30.48                                                    24.59                                                    28.24                                                    27.74                                                    13.21                                                    29.42                                                    24.70                                                    25.69                                                     21.00                                                    21.53                                                    26.53                                                    25.59                                                

Midjourney           69.41     12.59                                                     12.33                                                    13.90                                                    12.66                                                    12.71                                                    11.21                                                    13.83                                                    10.66                                                    12.79                                                    13.44                                                    9.43                                                      12.87                                                    14.68                                                    12.60                                                    12.87                                                

Dalle3               66.42     53.75                                                     62.43                                                    58.62                                                    65.88                                                    60.38                                                    59.34                                                    62.40                                                    33.95                                                    62.09                                                    54.73                                                    63.69                                                     52.88                                                    42.30                                                    55.87                                                    56.14                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     67.97     9.61                                                      7.80                                                     8.18                                                     8.19                                                     9.06                                                     8.04                                                     8.61                                                     8.54                                                     7.51                                                     7.66                                                     8.36                                                      8.25                                                     9.40                                                     7.64                                                     11.34                                                

Alt Diffusion        62.40     19.40                                                     16.30                                                    15.19                                                    16.02                                                    15.57                                                    15.30                                                    16.19                                                    16.49                                                    13.65                                                    12.01                                                    13.89                                                     14.46                                                    13.70                                                    14.08                                                    13.48                                                

Alt Diffusion m9     55.99     26.02                                                     19.62                                                    24.20                                                    26.24                                                    26.82                                                    25.70                                                    28.29                                                    28.33                                                    25.71                                                    15.44                                                    21.82                                                     20.15                                                    13.53                                                    22.08                                                    13.49                                                

Midjourney           69.41     15.85                                                     13.90                                                    13.85                                                    13.09                                                    13.76                                                    12.81                                                    12.52                                                    13.32                                                    13.02                                                    12.87                                                    12.65                                                     13.97                                                    11.71                                                    13.28                                                    11.61                                                

Dalle3               66.42     64.35                                                     49.36                                                    55.28                                                    58.83                                                    56.88                                                    61.65                                                    59.37                                                    61.20                                                    50.13                                                    26.19                                                    60.28                                                     50.97                                                    10.75                                                    56.98                                                    11.43                                                
 








Image Grounded Correctness (IGC) (%) across the different Indic languages in IndicTTI on the complete set.  

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     53.97     23.48                                                     23.57                                                    22.72                                                    22.93                                                    22.65                                                    22.49                                                    22.99                                                    23.40                                                    23.32                                                    23.25                                                    22.24                                                     22.63                                                    23.45                                                    22.37                                                    22.40                                                

Alt Diffusion        49.12     23.64                                                     24.30                                                    23.78                                                    25.12                                                    26.22                                                    23.45                                                    24.22                                                    22.43                                                    25.05                                                    25.54                                                    24.49                                                     24.41                                                    24.24                                                    24.35                                                    22.44                                                

Alt Diffusion m9     45.62     24.82                                                     28.39                                                    29.13                                                    29.85                                                    28.90                                                    30.43                                                    28.47                                                    23.49                                                    29.87                                                    27.94                                                    28.35                                                     25.72                                                    27.34                                                    28.45                                                    29.22                                                

Midjourney           52.72     22.54                                                     22.57                                                    22.63                                                    22.76                                                    22.03                                                    21.97                                                    23.10                                                    21.95                                                    22.79                                                    22.43                                                    22.63                                                     22.56                                                    24.10                                                    22.47                                                    22.12                                                

Dalle3               48.54     39.48                                                     45.13                                                    43.10                                                    45.86                                                    42.57                                                    43.72                                                    46.26                                                    30.42                                                    44.47                                                    40.34                                                    45.54                                                     38.23                                                    35.46                                                    41.71                                                    41.59                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     53.97     22.56                                                     22.23                                                    22.93                                                    23.06                                                    23.26                                                    22.83                                                    22.92                                                    22.83                                                    23.62                                                    23.01                                                    23.00                                                     22.91                                                    24.42                                                    22.93                                                    23.63                                                

Alt Diffusion        49.12     24.95                                                     23.69                                                    24.16                                                    24.64                                                    24.51                                                    24.45                                                    24.68                                                    24.93                                                    23.56                                                    23.53                                                    23.82                                                     23.78                                                    23.55                                                    24.13                                                    23.66                                                

Alt Diffusion m9     45.62     28.56                                                     25.87                                                    27.24                                                    28.05                                                    28.80                                                    28.25                                                    29.01                                                    29.06                                                    28.64                                                    23.28                                                    26.69                                                     25.31                                                    23.42                                                    26.73                                                    23.34                                                

Midjourney           52.72     24.05                                                     22.95                                                    23.04                                                    22.66                                                    23.25                                                    22.87                                                    22.76                                                    22.79                                                    22.60                                                    22.45                                                    22.66                                                     22.81                                                    21.08                                                    22.57                                                    22.39                                                

Dalle3               48.54     44.38                                                     38.97                                                    43.75                                                    43.64                                                    44.34                                                    44.87                                                    44.30                                                    44.88                                                    37.29                                                    28.82                                                    44.07                                                     38.84                                                    23.23                                                    43.55                                                    22.44                                                
 







Language Grounded Correctness (LGC) (%) across the different Indic languages in IndicTTI on the complete set of prompts. 

Model                en        c]@c@asm 
 (Beng)     c]@c@ben
 (Beng)     c]@c@guj
 (Gujr)     c]@c@hin
 (Deva)     c]@c@kan
 (Knda)     c]@c@mal
 (Mlym)     c]@c@mar
 (Deva)     c]@c@mni
 (Beng)     c]@c@npi
 (Deva)     c]@c@ory
 (Orya)     c]@c@pan 
 (Guru)     c]@c@san
 (Deva)     c]@c@snd
 (Arab)     c]@c@tam
 (Taml)     c]@c@tel
 (Telu) 
 Stable Diffusion     35.00     4.05                                                      4.34                                                     3.85                                                     4.26                                                     4.04                                                     4.62                                                     3.95                                                     4.15                                                     4.18                                                     4.00                                                     3.37                                                      4.03                                                     4.58                                                     4.59                                                     4.57                                                 

Alt Diffusion        32.47     6.31                                                      7.22                                                     6.81                                                     8.22                                                     9.95                                                     6.70                                                     7.21                                                     4.44                                                     8.34                                                     8.56                                                     7.47                                                      7.39                                                     7.14                                                     7.73                                                     6.71                                                 

Alt Diffusion m9     29.22     6.44                                                      11.48                                                    11.86                                                    13.70                                                    11.23                                                    13.16                                                    11.79                                                    4.32                                                     13.30                                                    10.69                                                    10.92                                                     8.76                                                     9.19                                                     11.41                                                    11.67                                                

Midjourney           34.11     2.60                                                      2.73                                                     2.94                                                     3.43                                                     3.24                                                     2.23                                                     3.46                                                     2.19                                                     3.65                                                     2.77                                                     2.65                                                      3.47                                                     4.28                                                     3.42                                                     2.47                                                 

Dalle3               33.04     25.79                                                     30.96                                                    27.64                                                    31.59                                                    28.51                                                    28.44                                                    30.62                                                    14.22                                                    29.77                                                    26.54                                                    31.37                                                     24.83                                                    18.57                                                    26.45                                                    26.52                                                
 Model                en        c]@c@urd
 (Arab)      c]@c@kas
 (Arab)     c]@c@kas
 (Deva)     c]@c@mai
 (Deva)     c]@c@awa
 (Deva)     c]@c@bho
 (Deva)     c]@c@hne
 (Deva)     c]@c@mag
 (Deva)     c]@c@sin
 (Sinh)     c]@c@brx
 (Deva)     c]@c@doi
 (Deva)      c]@c@gom
 (Deva)     c]@c@sat
 (Olck)     c]@c@snd
 (Deva)     c]@c@mni
 (Mtei) 
 Stable Diffusion     35.00     5.09                                                      4.61                                                     4.34                                                     4.34                                                     4.63                                                     4.22                                                     4.52                                                     4.41                                                     3.82                                                     3.90                                                     4.06                                                      4.17                                                     3.76                                                     4.16                                                     4.72                                                 

Alt Diffusion        32.47     8.17                                                      5.84                                                     6.69                                                     7.48                                                     7.44                                                     7.37                                                     7.59                                                     7.63                                                     6.16                                                     5.67                                                     6.63                                                      6.24                                                     5.84                                                     6.68                                                     5.71                                                 

Alt Diffusion m9     29.22     11.15                                                     7.80                                                     9.96                                                     11.18                                                    12.13                                                    11.24                                                    12.44                                                    12.52                                                    11.31                                                    4.97                                                     9.35                                                      7.59                                                     5.71                                                     9.28                                                     5.75                                                 

Midjourney           34.11     4.19                                                      3.15                                                     3.62                                                     3.29                                                     4.22                                                     3.30                                                     3.88                                                     3.43                                                     2.42                                                     3.32                                                     3.36                                                      3.35                                                     2.52                                                     3.47                                                     3.64                                                 

Dalle3               33.04     30.35                                                     23.55                                                    27.51                                                    28.71                                                    27.79                                                    30.15                                                    29.68                                                    29.73                                                    22.35                                                    11.46                                                    29.14                                                     23.96                                                    2.24                                                     27.91                                                    1.54                                                 
 










§ BENCHMARK RESULTS AND ANALYSIS

In this section, we report extended results on the common set and complete set of prompts.  The observations are consistent with those reported in the main paper on the common subset.

In the correctness-based metrics, the CLGC, IGC, and LGC metrics over the common set of prompts are reported in Tables <ref>, <ref>, and <ref>, respectively. Similarly, the results for the three metrics on the complete set of prompts are reported in Tables <ref>, <ref>, and <ref>. As observed in the main paper, across all the metrics, Dalle3 outperforms all other models when evaluated for Indic languages with a significant margin. Other models struggle to generalize on the Indic languages with AltDiffusion m9 performing the best among the other three models. This behavior is a result of increased generalizability due to the multilingual training of the model in 9 languages. On the other hand, while all models perform similarly for the English language, AltDiffusion m9 performs the worst, possibly due to catastrophic forgetting when trained for multilinguality.



In representation-based metrics, we evaluate the SCWL and DWL metrics on the complete set and observe that they follow the same patterns as their performance on the common set (refer Fig. <ref>). It is also observed that while AltDiffusion m9 has high distinctiveness across the concepts it generates, it provides poor self-consistency within the language, highlighting its instability and a tendency to seemingly generate diversely with or without relevance to the prompt. For the SCAL metric, the value obtained for AltDiffusion m9 on the common set of prompts comes out to be 21.47%, which is lower than the overall SCAL metric for the Stable Diffusion, AltDiffusion, Midjourney, and Dalle3 models is 25.44%, 23.73%, 26.75%, and 29.90%, respectively. This indicates an overall low consistency of AltDiffusion m9 in generating concepts across different languages.




§ QUALITATIVE ANALYSIS

In this main paper, we qualitatively analyzed the images generated by the different models across the different languages. We present more qualitative results here in Figs. <ref>, <ref>, and <ref> corresponding to Stable Diffusion, Alt Diffusion, and Midjourney, respectively.

The Stable Diffusion model (Fig. <ref>) generates images containing a high number of individuals, temples, flowers, and gods. In the case of Arabic scripts (kas_Arab, snd_Arab, urd_Arab), the model produces men wearing Muslim caps and women in burkhas or niqabs. For all Devanagri scripts as well as for as guj_Gujr in Gujarati script, the model generates individuals with sarees, and tilak which are often worn in the Indian culture. Additionally, it produces gods and temples. Sanskrit (sans_Deva) in particular, generates a large number of gods and temples due to its extensive religious context. This pattern is present across all languages using the Devanagri script. For languages using the Bengali script (asm_Beng, ben_Beng, mni_Beng), the model produces a significant amount of distinctive greenery. Sindhi in Arabic script (snd_Arab) is the only language generating a substantial amount of pornographic content. 



The AltDiffusion model (Fig. <ref>) generates images of couples, gods, places of worship, and occasionally of Indian monuments such as the Taj Mahal, especially for languages with the Devanagari script. When generating for Arabic scripts, such as kas_Arab, snd_Arab, and urd_Arab, the model generates images of men wearing Muslim caps, women in burkhas or niqabs, and mosques showcasing a correlation between the Arabic script and Muslim culture. In the Gurumukhi script (pan_Guru), the model depicts individuals with long beards and turbans, highlighting correlations between the script of the Punjabi language with stereotypical portrayal of people from Punjab. Languages from the Dravidian family, including Telugu (tel_Telu), Tamil (tam_Taml), Malayalam (mal_Mlym), and Kannada (kan_Knda), along with Sinhala (sin_Sinh), feature images of dark-skinned individuals, possibly associating skin-color with the individuals being generated.


In Midjourney (Fig. <ref>), for the Devanagari script, the model produces a variety of visuals, including women, gods, and deities in Hinduism (like Shiva, Ganesha, and Krishna), pandits (priests), temple-like structures (religious places of worship), elephants, and tigers (typically shown in stereotypical depictions of India). Within the Dravidian family languages such as Malayalam (mal_Mlym), Kannada (kan_Knda), Telugu (tel_Telu), and Tamil (tam_Taml), common elements include jewelry such as necklaces, forehead pendants, earrings, and dark-skinned individuals, particularly men. Bengali scripts (asm_Beng, ben_Beng, mni_Beng) often feature bridal women in red sarees adorned with jewelry.  Other commonly generated images include images of food, such as fish, which is prevalent in Bengali culture. Languages with Arabic script like Kashmiri (kas_Arab), Urdu (urd_Arab), and Sindhi (Snd_Arab) commonly depict women with hijabs or niqabs, men in headcovers, and places of Islamic worship such as mosques. In Gujarati, in addition to Devanagari influences, food-related imagery is prominent. Finally, Punjabi in Gurumukhi script (pun_Guru) frequently showcases individuals wearing turbans with a Gurudwara (place of worship in Sikhism, a religion commonly practiced by residents of Punjab) in the background. For certain languages such as Santali (sat_Olck) and Manipuri in Meitei script (mni_Mtei), Midjourney generates Asian women, depicting no Indian cultural influences (Fig. <ref>. It is interesting to note that these two languages also produce random outputs in the Dalle3 model, which understands many of the other Indic languages. 












 Paper Title: High-Quality, ROS Compatible Video Encoding and Decoding for High-Definition Datasets.





MUFASA: Multi-View Fusion and Adaptation Network with Spatial Awareness for Radar Object Detection
    Xiangyuan Peng1,2 Miao Tang3 Huawei Sun1,2Kay Bierzynski2Lorenzo Servadei1 Robert Wille1

    August 5, 2024
==================================================================================================

empty
empty



§ ABSTRACT 
Robotic datasets are important for scientific benchmarking and developing algorithms, for example for Simultaneous Localization and Mapping (SLAM). Modern robotic datasets feature video data of high resolution and high framerates. Storing and sharing those datasets becomes thus very costly, especially if more than one camera is used for the datasets. It is thus essential to store this video data in a compressed format. This paper investigates the use of modern video encoders for robotic datasets. We provide a software that can replay mp4 videos within ROS 1 and ROS 2 frameworks, supporting the synchronized playback in simulated time. Furthermore, the paper evaluates different encoders and their settings to find optimal configurations in terms of resulting size, quality and encoding time. Through this work we show that it is possible to store and share even highest quality video datasets within reasonable storage constraints. 






§ INTRODUCTION


Datasets are an important tool for research and development in mobile robotics. For example, for evaluating Simultaneous Localization and Mapping (SLAM) algorithms <cit.>, datasets play an important role <cit.>. Sensor technology is advancing rapidly, such that better camera sensors with higher resolutions and higher frame rates are available for use in robotic systems. While older datasets like KITTI from 2013 <cit.> use 1.4MP cameras with a frame rate of 10Hz, later datasets offer for example 5MP images with 10 Hz <cit.>. 

Our most recent mapping robot for collecting datasets features five pairs of stereo cameras, facing front, left, right, back and up, each with 5MP <cit.>, and a 16 node "Cluster on Wheels" <cit.> to collect this data with 60Hz. Additionally, that robot features other vision sensors (e.g. a 6-lens Ladybug 5+, stereo infrared, RGB-D, event camers) as well as 7 LiDARs. The datasets collected with this robot are valuable tools to evaluate the performance of SLAM algorithms. 

It is especially interesting to use this high-end data to then simulate lower quality data by resizing the images to get lower resolutions or dropping frames to simulate lower frame rates. This way, one can investigate in detail the effects of resolution and frame rate vs. accuracy, CPU and memory consumption. We have developed the SLAM Hive system <cit.> to do such benchmarking automatically in the cloud and provide an easy to use web interface for analysis of the data[SLAM Hive Demo: <https://slam-hive.net/>]. 

As discussed, high resolution and high-quality datasets are valuable for evaluating SLAM. The problem is with this data is, that it is really big. As can be seen in Table <ref>, the 11 5MP cameras with just 30Hz frame rate of the initial ShanghaiTech Mapping Robot Outdoor Dataset <cit.> has a raw data size of more than 2TB, the RGB size would be more than 7TB and even not-so-great quality 90 JPG would still be 446GB that would needed to be shared with the community. This is intractable in terms of transmission time over the internet, but also in terms of storage space for most computers. Therefore, the goal of this paper is to investigate how better compression rates can be achieved, while still maintaining excellent image quality, reasonable encoding times and small size. In order for the compressed dataset to be useful as a reference dataset for SLAM research and benchmarking, a truly excellent video quality has to be maintained. As reported in <cit.>, compression artefacts can in fact reduce the applicability of video streams to various applications such as object detection. Specifically, we do not need to encode the videos in real-time, but will do the compression in post-processing, because there are not enough computation resources even on a 16-node cluster to do the de-bayering and encoding in high-quality in real time. In the end our experiments will justify our choice of encoder and settings, which enables us to share the above mentioned video dataset with a size of just 43GB.

The contributions of this paper are:

	
  * Investigation of different high-quality compression/ encoding options for video data;
	
  * Experimental evaluations of the options based on a high-resolution, high-frame rate test video;
	
  * Development and open-sourcing [<https://github.com/STAR-Center/ros_mp4>] of a python tool to replay MP4 videos with yaml metadata, including support for synchronized replay with simulated time;
	
  * Evaluation of source frame rate effects on encoding.


The rest of the paper is organized as follows: Section <ref> introduces the background and related work. Section <ref> introduces our software framework for encoding and decoding the videos within the ROS 1 and ROS 2 middle-ware. Extensive evaluations regarding encoding algorithms and their settings vs. quality, encoding time and video size and further experiments are performed in Section <ref>  and conclusions are drawn in Section <ref>.




§ BACKGROUND






When collecting video datasets, several options for storage come to mind. In this paper we only consider color video data, as this is the most prevalent and information rich video format. High-quality computer vision cameras, such as the Grasshopper3 (GS3-U3-51S5C-C), provide raw data, that is bayered <cit.>. This means, that there are four channels of red, green (2x) and blue gray-scale values provided, typically each using one byte. After de-bayering, the image size is tripled, since every pixel now gets an RGB or YUV444 color value, interpolated over the surrounding, colored pixels, each using 3 bytes. Given the 5,013,504 pixels (2048x2448) of the 5MP camera and a 60Hz update rate, this results in 902MB per second or 3.3 TB per hour - for one camera stream. Thus compression is needed. One example frame is shown in Fig. <ref>.

Traditionally, robotics is concerned with real-time video compression for teleoperation <cit.>, robot-to-robot communication <cit.> or transmission of range data <cit.>. But the aim for this paper are datasets with no visible degradation in image quality. Lossless image compression approaches like PNG do not help much with real camera data, since the camera noise results in most pixels having different values even if they look similar. The next best option is to use JPEG. In Table <ref> we show that even very low quality jpeg values such as 75 result big file sizes. Thus real video encoders promise better video quality and smaller size.







 §.§ ROS and Video


Robotic datasets often use ROS, specifically ROS bags to store the data. Video data in ROS 1 is stored in image messages, one for each frame. The sensor_msgs/Image message type provides different encoding options, such as bayer_GBRG8, rgba8, yuv422 and OpenCV types  [<http://docs.ros.org/en/indigo/api/sensor_msgs/html/image__encodings_8h_source.html>]. The type sensor_msgs/CompressedImage, on the other hand, supports PNG, JPG and h264. ROS 2 also supports newer codecs, such as h265 [<https://github.com/ros-misc-utilities/ffmpeg_image_transport>]. 

An important feature of ROS is the time-synchronized playback with simulated time. For that ROS will take the time stamps of one bagfile and publish that time on the topic /clock. Other rosbag play instances will listen to that simulated time, if the parameter use_sim_time is set to true beforehand, and play their messages at the according, simulated times, thus achiving a synchronized playback of multiple bagfiles. 

In this project we decided to encode the video data of the dataset as mp4 files. This has several advantages: 1) Users can inspect the data very easily, by just playing them in their favorite video player (e.g. VLC). 2) We can use the newest encoders and are not held back by compatibility issues regarding provided encoders of the compressed image format. 3) Our playback tools work with ROS 1 and with ROS 2, so it is very easy to use the video dataset with either software.

To support the functionality similar to rosbag play, our software uses metadata, specifially the time stamps of the image frames, stored in an external yaml file. Using this we can replay the videos in ROS 1 and ROS 2 just as if they came from a rosbag, including support for time-synchronize playback via simulated time.

Next the different options for video codecs are presented.
 


  §.§.§ H.264/AVC


In 2004 the codec H.264/AVC was introduced <cit.>. It has numerous innovative features such as multi-reference frame motion compensation, variable block size motion compensation, and intra-frame prediction encoding.






  §.§.§ H.265/HEVC

The H.265/HEVC  codec was introduced in 2013 <cit.>. It further optimizes predictive coding techniques based on the inheritance of H.264/AVC, including more refined intra-frame and inter-frame prediction<cit.>, larger coding units<cit.>, and higher precision motion vectors<cit.>, significantly improving compression efficiency. 






  




  §.§.§ AV1

Following H.264/AVC and H.265/HEVC, the AV1<cit.> encoding standard was introduced in 2018. It features high compression efficiency and is open-source. AV1 upgrades the granularity of intra-frame directional prediction and boasts a more powerful inter-frame encoder<cit.>, significantly enhancing the accuracy of compound prediction.





 §.§ FFmpeg

FFmpeg<cit.> is an open-source multimedia framework supporting  numerous codecs, including H.264, H.265, and AV1.



Among the key parameters in FFmpeg's command line, -CRF (Constant Rate Factor) and -PRESET are crucial for controlling compression quality. CRF is a "constant quality" encoding mode that dynamically adjusts quantization parameters (QP) across frames to maintain a consistent visual quality. Users can adjust the quality of the output video by specifying a CRF value between 0 and 51, where a lower value corresponds to higher video quality and a larger file size.The -PRESET parameter balances encoding speed and compression efficiency, offering options ranging from "ultrafast" to "placebo", allowing users to choose the appropriate encoding speed based on specific application scenarios and hardware performance.




 §.§ Image Quality Measurement



To ensure the visual quality of videos, conducting detailed subjective and objective evaluations of video quality is crucial. VMAF (Video Multi-Method Assessment Fusion) is a  video quality assessment tool jointly developed by Netflix and the academic community <cit.>. 

VMAF measures video quality through integrating multiple visual quality indicators, generating objective scores that highly align with human visual perception. Based on the nuSvr algorithm of SVM, VMAF first extracts key visual features in the evaluation process, including but not limited to Visual Information Fidelity (VIF), Detail Loss Metric (DLM), and Temporal Information (TI), comprehensively considering the spatial and temporal dimensions of the video. Then, VMAF utilizes a pre-trained SVM model to combine these features with human subjective scores, assigning different weights to each video feature. A score is generated for each frame, and the final score of the video is calculated using the mean of those values. 

Table <ref> shows details of an I-frame of our test video encoded with different parameters, thus resulting in different VMAF scores. There are sill some differences in the frames visible for VMAF scores below 99.50. But for scores higher than that no difference can be made out with the naked eye. Therefore we chose 99.50 as our target VMAF score for highest-quality video encoding. 



§ ROS MP4 ENCODING AND PLAYBACK FRAMEWORK





In this section, we will delve into the detailed steps of compressing rosbag and playback. The camera image messages recorded in rosbag are essentially a series of consecutive image frames with timestamps and other metadata, which is quite similar to the characteristics of video streams. 

The first step is to separate the image frames from their associated timestamps and other metadata. The camera images recorded in rosbag may not be directly stored in common BGR formats, but may use formats like "bayer_gbrg8" or may have already undergone some form of compression during recording. In such cases, necessary format conversion and preprocessing may be required. Once the image frames are extracted, we can utilize  FFmpeg to compress these image frames into the desired video format.


 §.§ Yaml File

Apart from image frames, other metadata (such as timestamps, frame IDs, intrinsic parameters, etc.) also need to be properly preserved to ensure that we can fully restore all the original information from the rosbag messages. To achieve this, we use the YAML file format to store these metadata.  For typical visual rosbag datasets, there are two key topics: "image_raw" (or a similar name) for storing raw image data, and "camera_info" containing camera calibration and distortion information related to each frame. After analyzing the information structure of these two topics, we find that the information in the "camera_info" topic actually contains some key information corresponding to the image frames in the "image_raw" topic, such as frame_id and timestamp. Therefore, we define the structure of the YAML file as follows:








































 §.§ Play Back

To play back this compressed dataset composed of YAML file and video, we have developed a powerful script. This script not only integrates and publishes topic information corresponding to the original rosbag dataset, but also retains many of the same features as playing a rosbag. By parsing the timestamp information in the YAML file and utilizing ROS's clock mechanism, our script can accurately simulate the playback time of the dataset in a real-world environment, ensuring that the content of the published topics remains consistent with the original rosbag playback in terms of both timing and content. This high level of accuracy provides users with an experience almost identical to directly playing a rosbag. Fig. <ref> shows the different playback options available. 

















§ ENCODING EXPERIMENTS





To determine the optimal compression combination for video quality and compression efficiency for the new dataset structure, we carefully selected a 1-minute clip from the recorded dataset that contains both static and dynamic scenes as a test sample. This clip combines a 5-second static outdoor scene and a 55-second dynamic outdoor scene, which can represent the typical characteristics of the dataset. The video was recorded on the ShaghaiTech Mapping Robot with a Grasshopper3 (GS3-U3-51S5C-C) camera with 2048x2448 pixel at 60Hz frame rate. 



For evaluation we utilize the 60 second test video, but then extrapolate the results to a dataset of 14 such video streams each 2 hours long, as this is the expected dimension of our future ShanghaiTech Mapping Robot Datasets. The robot features 11 such datastreams at full frame rate plus the LadyBug5+ camera with 6 streams at half the frame rate, so in total around 11 + 6/2 = 14. The results help us to understand the computation and storage demands of a dataset compressed with the according settings.









To accurately evaluate video quality, we first converted the selected rosbag clip to YUV444 format, a format commonly used in video editing due to its ability to preserve the original quality of the video well. We used the YUV444 format video as the testing ground truth to accurately measure the quality of the compressed video. To convert from the raw Bayer patter GBRG8 to YUV444 we used the debayering algorithm R and B Interpolation, as presented in <cit.>,  to get the BGR image matrix, and then used FFmpeg to convert it to the YUV444 format.


The following experiments all utilize CPU encoding on an Intel Xeon(R) Gold 5218R CPU and 96GB of RAM. There are also encoding blocks included in many GPU cards. But those are all geared towards real-time encoding, e.g. for video streaming, and sacrifice quality for speed.
<cit.> reports that for mobile robotics, if real-time encoding is required, hardware GPU encoders are the best. But they also find that the highest quality encoding is done by software encoding. The aim for this work is to find options to encode videos in very high quality, with no visible compression artifacts. For this we require the VMAF score of the compressed videos to be at least 99.40. 


In our experiments we selected the CPU codecs x264, x265, and svtav1, and conducted extensive tests using the FFmpeg tool within the range of -crf 16 to 25 and -preset placebo to medium. Tables <ref>, <ref> and <ref> show the different encoding options and the resulting bitrates, sizes and VMAF of the 60s test video for svtav1, x264 and x265, respectively. The tables are visualized in Fig. <ref> and Fig. <ref> for VMAF vs bitrate and encoding time, respectively. 

 As shown in Fig. <ref>, our test data indicates that svtav1 consistently outperforms the other two codecs. Specifically, the VMAF scores of x264 and x265 only approach 99.50 when the -preset is set to slow and veryslow, while 55% of svtav1's VMAF scores fall between 99.40 and 99.60, significantly outperforming the other two codecs.


 60s video compression times greater than 1000 seconds would result in compression times of more than 9.7 days for our 14 stream 2h dataset, which exceeds our patience for video post-processing. Therefore, we further narrowed down our analysis to codec settings combinations with VMAF scores between 99.50 and 99.60 and preset values greater than 4, as shown in Table <ref>.

Analysis of the Table <ref> reveals a compression combination resulting in a good VMAF score: -CRF 19, -preset 6. Given that our primary concern is compression effectiveness and quality, and according to Table <ref>, there is no visual difference between VMAF scores of 99.50 and 99.55, we ultimately chose the compression combination of -CRF 22 and -preset 6. This combination achieves the best balance in terms of compression effect, quality, and time consumption.

















































































 §.§ Effect of Surce Framerate

The maximum framerate of our dataset is 60Hz. But occasionally we may also opt to collect lower frame rate data. The question this experiment is answering is: What is the best encoding setting for lower frame rates than 60Hz? The hypothesis is, that higher frame rate videos feature more similar image content, so encoding may be more efficient for those streams. On the other hand, camera data is inherently noisy, such that, at least for very high quality videos, no practical difference may be found.

To answer this question we subsample our one minute test stream by taking only every n-th frame, to get to the frame rates shown in Table <ref>. For a fixed VMAF of about 99.50, we then record the average frame sizes and encoding times per frame.

The results show that indeed higher frame rates encode each frame more efficiently. But, since the settings kind of set a goal bitrate, given our goal VMAF, for lower frame rates, lower settings can be chosen. The table shows good settings for very high quality encoding at lower frame rates with acceptable sizes and encoding times. 











 §.§ Compression Results of "ShanghaiTech Mapping Robot Outdoor Dataset"



The practical test of this paper comes from the Mapping Robot Outdoor Dataset <cit.>. This is a ground-based unmanned vehicle dataset that includes multiple sensors. In particular, we focus on the recorded data from two types of RGB cameras provided by the dataset: FLIR Grasshopper3 (GS3-U3-51S5C-C) and FLIR LadyBug5+. They provide high-resolution and high-frame-rate recorded data. We tested our dataset scheme on the data collected by these two cameras, as shown Table <ref>. For this dataset we ran with half the maximum framerate. The results show that compared with the original data, the dataset scheme using video compression technology can save 2,436.5GB of space.








§ CONCLUSIONS



This paper presented a method for encoding robotic dataset video data as mp4 files. Our software allows the replay of mp4 files within the ROS framework, as if the video was stored in a ROS bagfile, including timestamps and replay with simulated time. This enables us to utilize modern video codecs like av1 and is also user friendly, because the data can be viewed with any video player and the data is compatible with ROS 1 and ROS 2 without the need for conversion. The paper furthermore evaluated which codec and setting is the best for very high video quality without any visible compression artifacts, while maintaining reasonable compression size and encoding time and finally selected an av1 setting. This encoding is more than ten times smaller and with higher quality than JPG encoding with relatively high quality 90. We also explored the effect of source frame rate on the encoding and found that higher frame rates encode frames more efficiently due to higher similarity in the frames. With those frame rate experiments we also suggested settings for high-quality encoding with lower source frame rates. In our ShanghaiTech Mapping Robot paper we then applied those results to a real, big robotic dataset.

In the future we expect more datasets from the ShanghaiTech Mapping Robot, even with 60Hz frame, to be released to the public.






unsrt








 Paper Title: Regional quality estimation for echocardiography using deep learning.



Generalised BBGKY hierarchy for near-integrable dynamics
    Jacopo De Nardis
    Submitted August 5, 2024
========================================================

§ ABSTRACT 
Automatic estimation of cardiac ultrasound image quality can be beneficial for guiding operators and ensuring the accuracy of clinical measurements. Previous work often fails to distinguish the view correctness of the echocardiogram from the image quality. Additionally, previous studies only provide a global image quality value, which limits their practical utility. In this work, we developed and compared three methods to estimate image quality: 1) classic pixel-based metrics like the generalized contrast-to-noise ratio (gCNR) on myocardial segments as region of interest and left ventricle lumen as background, obtained using a U-Net segmentation 2) local image coherence derived from a U-Net model that predicts coherence from B-Mode images 3) a deep convolutional network that predicts the quality of each region directly in an end-to-end fashion. We evaluate each method against manual regional image quality annotations by three experienced cardiologists. The results indicate poor performance of the gCNR metric, with Spearman correlation to the annotations of ρ=0.24. The end-to-end learning model obtains the best result, ρ=0.69, comparable to the inter-observer correlation, ρ=0.63. Finally, the coherence-based method, with ρ=0.58, outperformed the classical metrics and is more generic than the end-to-end approach.


    Keywords:


Cardiac segmentation, Ultrasound, image quality, Coherence, Signal-To-Noise Ratio






§ INTRODUCTION


Image quality is one of the main challenges in ultrasound imaging and can differ significantly between patients and imaging equipment. In echocardiography, many factors influence image quality such as the ultrasound scanner, the patient, and the probe. Several quantitative measurements using the images are performed. However, this requires image quality good enough for the given measurement. Different measurements have different image quality requirements, for instance, left ventricular (LV) volume, ejection fraction (EF), and strain measurements require good image quality in the entire myocardium. On the other hand, mitral annular plane systolic excursion (MAPSE) only requires good image quality in the annulus. Good image quality should generally provide measurement values with low uncertainty. Estimating image quality can be useful in the following ways:

    
  * To guide operators to achieve as good image quality as possible while scanning.
    
  * To automatically select the best images, recordings, and the best cardiac cycles to use for a given measurement.
    
  * As quality assurance, e.g. to warn the user when an image is not good enough for a measurement, and to automatically approve/disapprove individual myocardial segments based on image quality.
    
  * In data mining projects, to exclude cases with insufficient quality for reliable measurements.


We distinguish between two types of quality of ultrasound images: view quality/correctness and image quality. 
In this work, we will focus on image quality specifically. For view correctness, previous work has demonstrated that 3D ultrasound can serve as training data to automatically identify the transducer rotation and tilt in relation to the desired standard view and can guide the user to the correct position <cit.>. 
For image quality, the classic ultrasound signal-processing metrics are the contrast ratio (CR) <cit.>, contrast-to-noise ratio (CNR) <cit.>, and generalized CNR (gCNR) <cit.>. These three metrics need a region of interest (ROI) and a background region to compare against. More recently, global image coherence (GIC) <cit.> has been proposed as a general quality metric that does not require the selection of these two regions. The image coherence measures how well the signals of the transducer elements align after delay compensation, with more alignment corresponding to clearer and sharper images. From the above mentioned methods, only the GIC can be used directly and automatically for measuring image quality separately as it does not require selecting an ROI and noise region. However, this approach requires channel data, which is not readily available in practice and does not give regional metrics.  

Several automatic methods for measuring ultrasound image quality have been published applicable to cardiac imaging. Abdi et al. <cit.> used a recurrent neural network to predict the global quality of cardiac cine loops. The criteria for quality assessment take both image quality and view correctness into account. In subsequent studies <cit.>, they used an architecture that performs both view classification and global quality prediction simultaneously. The image quality metric is a global criterion based on the manual judgment of the clarity of the blood-tissue interface. Labs et al. <cit.> used a multi-stream neural network architecture where each stream takes in a sequence of frames and predicts a specific quality criterion. The criteria are global and take both view correctness and image quality into account. Karamalis et al. <cit.> detect attenuated shadow regions with random walks resulting in a pixel-level confidence map. Unlike the other methods above, this method is not based on deep learning. It provides a local, pixel-level metric, but it only measures the visibility of regions and not the quality of their content. 



All of the automatic methods mentioned above have the limitation that they only provide a global image quality evaluation and/or do not assess the image quality separate from the view correctness.  




§ METHODS

In this work, we developed and compared three fully automatic methods to asses regional image quality in cardiac ultrasound separate from the view correctness:

    
  * Classical ultrasound image quality metrics, such as contrast ratio and contrast to noise ratio, applied in cardiac regions automatically extracted using deep-learning segmentation.
    
  * Deep-learning predicted ultrasound coherence, which is a measure of how coherent a signal is received by the transducer elements, together with deep-learning segmentation.
    
  * End-to-end prediction of regional image quality.


The rest of this section first presents the datasets used to develop these methods, and then presents each of the three methods.



 §.§ Datasets




  §.§.§ VLCD


The Very Large Cardiac Channel Data Database (VLCD) consists of channel data from 33280 frames from 538 recordings of 106 study participants <cit.>. It contains parasternal short axis (PSAX), parasternal long axis (PLAX), apical long axis (ALAX), apical two-chamber (A2C), and apical four-chamber (A4C) views. We split the VLCD dataset on the study participant level into train, validation, and test sets, 70%, 15%, and 15% respectively. 



  §.§.§ HUNT4


The Nord-Trøndelag Health Study dataset (HUNT4Echo) is a clinical dataset including among others PSAX, PLAX, ALAX, A2C, and A4C views. Each recording contains 3 cardiac cycles. We use two subsets of the HUNT4Echo dataset.

    
  * Segmentation annotation dataset A fraction of 311 study participant exams, the segmentation annotation set <cit.>, contains single frame segmentation annotations in both ED and ES as pixel-wise labels of the left ventricle (LV), left atrium (LA), and myocardium (MYO) in ALAX, A2C, and A4C views.
    
  * Regional image quality dataset For this work, we created an additional dataset of image quality labels. The local image quality labels are manual annotations that asses the image quality of the cardiac regions of interest on a subset of the HUNT4 dataset in ALAX, A2C, and A4C views. Section <ref> describes the annotation process in more detail.












 §.§ Regional image quality annotation on HUNT4
  
An annotation tool was developed specifically for this project using the open-source Annotation Web software[<https://github.com/smistad/annotationweb>] <cit.>.
The tool was made to enable clinicians to annotate regional image quality as efficiently and accurately as possible. The tool is freely available and can be adapted to other image quality projects. The image quality annotations were performed by three experienced cardiologists using the following protocol:


  * Annotate the end-diastole (ED) and end-systole (ES) frame of each recording, and optionally other frames if the image quality changes significantly during the recording.

  * If the majority of the cardiac regions of interest is out-of-sector, label it as out-of-sector. Otherwise, label the part of the region that is inside the sector according to the definitions in Table <ref>. We ignore the out-of-sector regions in the remainder of this work.

For the first round of annotations, each of the three clinicians annotated the same 10 frames from 5 recordings of 2 study participants. We used this dataset to calculate the inter-observer variability. For the second round of annotations, the three clinicians collectively annotated 458 frames from 158 recordings of 65 study participants. The annotations from the second round form the regional image quality dataset. This dataset was split randomly at the study participant level into train, validation, and test sets, allocating 70%, 15%, and 15% of the data to each set respectively.








 §.§ Regional image quality estimation









  §.§.§ Classical ultrasound image quality metrics

For the classical image quality metrics, deep-learning segmentation is used to extract the annulus regions and each of the myocardial segments as regions of interest and the LV as the background region. Appendix <ref> gives more details about the procedure for dividing the segmentation into regions. The four classical ultrasound image quality metrics below were tested.
We apply histogram matching <cit.> to a Gaussian distribution (μ=127 , σ=32) for the B-Mode grayscale images before calculating pixel-based quality metrics.




  
  • Pixel intensity is the average pixel intensity value in each region.


  
  • Contrast Ratio (CR) <cit.> is defined as 
  
    CR=μ_segment/μ_LV

  where μ_segment is the average intensity in each region and μ_LV is the average intensity inside the LV lumen.
  
  
  • Contrast to Noise Ratio (CNR) <cit.> is defined as 
    
    CNR=μ_segment-μ_LV/√(σ_segment^2+σ_LV^2)

    where σ_segment is the standard deviation in each region and σ_LV is the standard deviation inside the LV lumen.

  
  • Generalized CNR (gCNR) <cit.> is defined as the maximum performance that can be expected from a hypothetical pixel classifier based on intensity using a set of optimal thresholds. It is calculated as 

    gCNR= 1-1/2∑_i=0^MAX_imin{p_segment(i), p_LV(i)}

where p_segment(x) is the probability density function of the pixel intensities inside the region the gCNR is calculated for, p_LV(x) the probability density function of the pixel intensities inside the LV lumen, and MAX_i the maximum possible pixel intensity. 
 Fig. <ref> shows an example of the probability density functions for one of the regions as ROI and the LV lumen as background.








  §.§.§ Local, deep-learning predicted image coherence as quality metric
 

We use the VLCD dataset to calculate the coherence factor <cit.> for each pixel in the ultrasound image. This factor is the ratio between the amplitude of the sum of the received signals to the sum of the amplitudes of those signals, 

    CF = ∑_n=1^NS_i/∑_n=1^N|S_i|

where S_i is the delayed signal for the i-th transducer element. This is equivalent to taking the coherent sum of the signal and dividing it by the incoherent sum of each signal. Thus, the coherence factor measures of how well the complex signals of all transducer elements align. The remainder of the signal-processing chain is the native processing of the GE HealthCare Vivid E95 system[ Gundersen et al. <cit.> describe this signal-processing chain in more detail.] but without the log compression. The result is a coherence image with the same dimension as the B-Mode image.  The final preprocessing step applies gamma normalization with γ=0.5 on the coherence images,

    t_i,normalized=t_i^γ

where t_i are the pixels of the target coherence image. The corresponding B-mode images are generated from the channel data using the same, native signal-processing pipeline. 



The HUNT4 dataset, like most ultrasound datasets, does not include channel data. Therefore, VLCD was used to train an image-to-image network that takes as input the grayscale B-mode image and predicts the coherence image, which is then used to calculate local image coherence. We use a lightweight U-Net architecture inspired by the U-Net 1 architecture in <cit.>, with characteristics listed in Table <ref>. As coherence is related to image quality, we only apply augmentations that do not influence the quality of the image. Furthermore, the coherence should be invariant to different gain settings, so we additionally augment with brightness adjustments on the B-mode image while keeping the target coherence image unchanged. During training and validation of the coherence prediction model, we sample a random frame from each recording in the train and validation set respectively during each epoch. During testing, we use all frames in the test set. The local image coherence quality metric of a region is the average pixel value of all pixels corresponding to the region in the coherence image. This is the same as the pixel intensity metric above but applied to the coherence image instead of the B-Mode image.









  §.§.§ End-to-end deep-learning quality prediction
 

The end-to-end learning approach trains a convolutional neural network on the regional image quality dataset to predict the quality of each region directly. We treat the problem as a regression task where the model predicts a score for each segment. Table <ref> shows the correspondence between quality scores and annotation labels. As architecture, we use a MobileNetV2 <cit.> that predicts the image quality labels of all regions simultaneously. Appendix <ref> describes the ablation study conducted to justify this setup. 








§ EXPERIMENTAL SETUP





 §.§ Evaluation of coherence prediction from B-mode


We use the structural similarity index (SSIM) <cit.>, peak signal-to-noise ratio (PSNR)[The maximum pixel value for coherence images is 1.], and relative pixel error (RPE) to evaluate the coherence image prediction network. We define the RPE as 

    RPE = |t_i-p_i|/max(t_i,ϵ)

where t_i are the pixel values of the target coherence image, p_i the pixel values of the predicted coherence image, and ϵ=1e-4. Section <ref> contains the results of this experiment.



 §.§ Evaluation of quality metrics



The correlation and accuracy of each quality metric were measured by comparing them to the expert annotations on the test set of the regional image quality dataset. For the classic image quality methods and regional coherence method, linear regression models were used to map the quality metric values to image quality labels. The train and validation set were used together to fit the linear regression model and evaluate it on the test set. Section <ref> contains the results of this experiment.



 §.§ Comparison to inter-observer variability




We compare the end-to-end, local image coherence, and gCNR-based model to the inter-observer variability on the data obtained in the first round of annotation, explained in subsection <ref>. For the local image coherence and gCNR-based models, the same linear model as subsection <ref> was used to map the metrics to quality labels. The inter-observer variability is calculated from the aggregate of the three unique pairwise score errors between each of the three annotators:

    e_inter-observer = e_12∪ e_23∪ e_13

where e_ij is the score difference between operator i and j. The error metrics of the automatic methods are calculated from the aggregate of the pairwise score errors between the output of the method and each of the three annotators:

    e_M = e_1M∪ e_2M∪ e_3M

where e_iM is the score difference between operator i and method M. 
Section <ref> contains the results of this experiment.






 §.§ Relation to variability in clinical measurements


This experiment evaluates whether there is a relation between the predicted quality and the agreement between different methods for clinical measurements. The hypothesis is that with lower image quality the variability, and thus the uncertainty, of the measurements between methods and between experts increases. More specifically,
this analysis compares peak global longitudinal strain (GLS) and ejection fraction (EF) measurements obtained either fully automatically with AI tools or manually by using GE HealthCare EchoPAC software on HUNT4 <cit.>. For AI estimation of GLS and EF, the deep-learning methods proposed by Østvik et al. <cit.> and
Smistad et al. <cit.> were used respectively. The study participants in HUNT4 used for model development were excluded from the analysis. For GLS, the predicted quality is the average quality of all segments over the full recording.
For EF, the predicted quality is the average quality of all segments in the end-diastole (ED) and end-systole (ES) frames of all cycles in the recording. Section <ref> contains the results of this experiment.












§ RESULTS






 §.§ Results of coherence prediction from B-mode
 

Table <ref> summarizes the average metric values on the test set. Fig. <ref> shows an example of the best, median, and worst-case predictions according to the relative pixel error. Fig. <ref> illustrates how the predicted coherence images are almost independent of the brightness/gain and contrast/dynamic range of the input B-Mode images. The main finding is that the difference between the estimated and ground truth coherence images is small and thus the predicted coherence images can be used to obtain coherence-based quality metrics for B-mode for which the channel data is not available. 























 §.§ Results of quality metrics
 

Table <ref> summarizes the results of the evaluation of the quality metrics. Fig. <ref> shows box plots of the quality metrics per image quality label for the end-to-end, coherence, gCNR, and intensity models. Fig. <ref> shows examples of B-mode images with varying quality together with labels from the annotators and automatic quality metrics. The main finding is that the end-to-end model performs the best, followed by the local image coherence metric. The classical ultrasound image quality metrics perform poorly.
















 §.§ Results of comparison to inter-observer variability
 

Fig. <ref> shows the bar plot comparing the automatic methods to the inter-observer variability and Table <ref> lists the corresponding average metric values. Using the Wilcoxon signed-rank test <cit.> and a significance level of p=0.05, we find that the difference in Mean absolute error (MAE) between each of the methods is statistically significant. The difference between the inter-observer MAE and the MAE of each of the methods is also statistically significant, i.e. the inter-observer MAE is higher than the MAE of the end-to-end model and lower than the MAE of the other two models.












 §.§ Results of relation to variability in clinical measurements
 

Fig. <ref> shows box plots visualizing the agreement between the measurements obtained automatically and with EchoPAC for each predicted quality category. The standard deviations in these plots represent how well the AI estimates agree with the manual references. The main finding is that the limits of agreement are narrower for higher qualities.












§ DISCUSSION




 §.§ Challenges and considerations



Assessing image quality based on human perception is inherently a subjective task, even when supported by clear definitions of image quality categories. This creates challenges for training and evaluation as there is no ground truth as the reference labels are a subjective estimation themselves. Therefore, it is not realistic to expect the automatic models to agree with reference labels as well as on tasks with well-defined ground truth labels. This, together with a rather fine scale of image quality categories, explains the low accuracies in Tables <ref> and <ref>. Fig. <ref> shows how the end-to-end model has on average less error than the annotators between each other. This indicates the model has learned to produce quality labels that strike a middle ground between the subjective assessments of the annotators. 

The end-to-end learning model overestimates low-quality regions and underestimates high-quality regions, as can be seen in Fig. <ref>. This means that the model can only explain a limited amount of variability in the image quality labels and is a result of minimizing the mean squared error (MSE) while dealing with subjective, and thus noisy reference values with fixed boundaries. We can eliminate this effect by fitting a linear model on the validation set that maps predicted image quality to image quality labels and applying it when doing inference on the test set. This increases MSE but gives more uniform performance over the image quality labels.

One reason for the weak correlation between the annotations and the pixel-based methods is the rough selection of ROI and background region. Fig. <ref> shows how the average metrics of the classic pixel-based and coherence metrics increase for each quality label until the good label, and then drop again for excellent.  This is because on the one hand in these high-quality images, the blood speckles can be visible inside the LV lumen, which is used as background region, and on the other hand the myocardium tissue, which is used as ROI, is less blurred resulting in a smaller spread of pixels with high intensity. This can be seen for the anterolateral wall and apex in the rightmost column of Fig. <ref>. One possibility to only select regions belonging to the tissue is to perform automatic pixel selection methods like Otsu thresholding <cit.> or percentile filters, but in our experiments this reduced the performance even further. 





 §.§ Design choices


The different methods in this study have a trade-off between accuracy and versatility. The default end-to-end network gives the best results but requires specific image quality labels for the task. Next, the coherence-based method is more generic and can potentially be applied more generally without the need for view-specific image quality annotations. Rindal et al. <cit.> showed that the GIC is not significantly different between apical views, but is higher for apical views than parasternal views. Thus, while a single image-to-image model can learn to predict coherence for different views, the mapping from coherence to image quality should be done for each group of views separately. Another advantage is that coherence can be used to give a global image quality metric without the need for a segmentation model. Finally, the pixel-based methods can be applied automatically in the most general way given a segmentation model to select ROI and background regions but also give the lowest accuracy. 

The ablation study of the end-to-end learning model showed that increasing the complexity of the model did not improve the performance. This is a result of the relatively small dataset size and the specific task of the model. For a more general model of image quality prediction with more varied input, e.g. one model for all cardiac views, a larger dataset and more complex model may be required.



 §.§ Clinical use


Image quality estimation can be the first step towards a method for giving reliability estimates to clinical measurements and quality control of fully automatic methods. Fig. <ref> shows that the variability in clinical measurements goes down with higher predicted quality. However, image quality is only one source of variability, so a reliability model would also need to include view correctness and other factors that determine whether a given input is difficult to assess. 

More direct use cases of the quality prediction model include the automatic selection of the best frame to perform a clinical measurement when multiple options are available, data cleansing in data mining, and automatic disapproval of segments for regional strain analysis. All the methods explored in this work are computationally efficient and can be run in real-time while scanning, and can thus be used as a guidance tool to enable clinicians to acquire images with better image quality.




§ REAL-TIME DEMO APPLICATION


To showcase the functionality of the end-to-end, real-time quality network, a real-time application was developed using the FAST framework <cit.>. The demo is a split-screen application that shows the B-Mode input to the left and the segmentation regionally color-coded by the quality as predicted by the end-to-end network to the right. Fig. <ref> provides a screenshot of the application in use. We provide a demo video <cit.> illustrating the application in action while a clinician operates a GE Vivid E95 scanner. The video can be accessed at <https://doi.org/10.6084/m9.figshare.26413984.v2>.






§ CONCLUSION


In this work, we developed and compared different deep-learning methods for regional image quality estimation in cardiac ultrasound. We show that classic pixel-based methods, such as (g)CNR, together with automatic image segmentation, give low agreement with the quality assessment of cardiologists. We developed a U-Net model to predict the coherence factor for each pixel in the ultrasound image and showed that the resulting coherence image can be used to assess the image quality in a pixel-based way with better performance than the classic measures. The best results, below inter-observer variability, are obtained by using an end-to-end deep-learning model. Finally, we show higher predicted quality is associated with lower limits of agreement between fully automatic and manual methods for the clinical measurements EF and GLS. 





IEEEtran







§ EXTRACTION OF CARDIAC REGIONS OF INTEREST
 


We use the nnU-Net <cit.> architecture to segment the cardiac images. The nnU-Net is used out of the box using the default configuration but without the final ensemble step. Instead, we train and validate on a single predefined 80% train, 10% validation, and 10% test split from the HUNT4 segmentation annotation dataset. Table <ref> summarizes the characteristics of the nnU-Net architecture. This model is described in more detail and compared to other segmentation models in our previous work <cit.>. We use two nnU-Nets, one for apical two- (A2C) and four-chamber (A4C) views, and one for apical long axis (ALAX) views. The nnU-Net for A2C and A4C views segments the left ventricle (LV), left atrium (LA) and myocardium (MYO). The nnU-Net for ALAX views additionally segments the aorta (AO).






    


The segmentation of the MYO is divided into eight regions using the following algorithm:

  
  * Extract the annulus points. For A2C/A4C views, these are the points where the MYO meets the LA. For ALAX views, these are the points where the MYO meets the LA and AO. Points A and B are the annulus points in Fig. <ref>.
  
  * Extract the apex of the LV, defined as the furthest points from the base points within the lv lumen. This is point C in Fig. <ref>.
  
  * Divide both the left and right part of the endocardium border, defined as the border between the LV and MYO regions, into three parts with equal length. This gives points D, E, F and G in Fig. <ref>.
  
  * Find the closest points on the outer MYO border for points C, D, E, F and G. These are points H, I, J, K and L in Fig. <ref>.
  
  * Fill in the regions by connecting the points via the contour, resulting in the MYO divided into six regions. 
  
  * Draw circles[Due to the unequal pixel spacing in depth and width, the annulus regions become ovals in the 256x256 segmentation maps. When plotting the images with equal spacing in width and depth, these regions become circles again.] with a radius of 2  millimeters around the annulus points, points A and B in Fig. <ref>. 
  We use these additional two regions to asses the local image quality of the annulus points in the image. The result are the eight regions, as in Fig. <ref>.
  
  * Remove any parts of regions that fall outside of the sector. The apical top regions in Fig.<ref>, i.e., the yellow and white masks, are examples of this. If more than 50% of all pixels inside the region fall outside the sector, we exclude the region from analysis.

The goal is to automatically quantify the image quality in each of these eight regions. The LV lumen is used as background region.








§ ABLATION STUDY OF THE END-TO-END LEARNING MODEL



In the ablation study, we evaluated the impact of modifying the architecture of the end-to-end learning model. Three different network architectures were tested: Cardiac View Classification (CVC) network <cit.>, MobileNetV2 <cit.>, and EfficientNet <cit.>. 
We tested approaching the problem both as a classification and regression task, with only the final dense layer and loss function being changed accordingly. Additionally, three basic network attention variations were tested using the automatic segmentation output. The default model did not use any attention and predicts each label directly, as in Fig.<ref>. For the other two versions, the region masks extracted from the segmentation were dilated with a square dilation filter of size 50x50 pixels and used as an additional input to the networks which then predict the label of one region at a time. The dilation filter reveals the direct vicinity around each region so the boundary between tissue and background becomes visible. The first variant used this dilated mask as hard attention by blacking out the other parts of the image, as shown in Fig. <ref>. For the second variant, the masks were used as soft attention as input to a side branch of the network, as proposed by Eppel <cit.>. For this version, we created an attention map and added it element-wise to the output of the first layer, corresponding to version 'c' in Eppel <cit.>. Fig. <ref> shows this configuration. 






The ablation study consists of two parts. Both parts used the training configuration listed in Table <ref> and data from the regional image quality dataset. In the first part, we examined the effect of changing the convolutional backbone and the effect of framing the problem as a classification or regression task. Table <ref> compares the predictions with the annotations on the test set for the different configurations using the default end-to-end model. In the second part of the ablation study, the backbone was fixed to MobileNetV2 <cit.> and the problem was framed as a regression task. Next, the different variants shown in Fig. <ref> were compared. Table <ref> summarizes the results on the test set. The default end-to-end model with no attention, the MobileNetV2 <cit.> architecture, and the problem framed as a regression task gave the best results and were thus used for this paper.









 Paper Title: StyleRF-VolVis: Style Transfer of Neural Radiance Fields for Expressive Volume Visualization.

                     















1391










IEEE VIS Paper








Representations & Interaction










  
  
  * K. Tang and C. Wang are with 
the Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556, USA.
E-mail: {ktang2, chaoli.wang}@nd.edu.



In volume visualization, visualization synthesis has attracted much attention due to its ability to generate novel visualizations without following the conventional rendering pipeline. However, existing solutions based on generative adversarial networks often require many training images and take significant training time. Still, issues such as low quality, consistency, and flexibility persist. This paper introduces StyleRF-VolVis, an innovative style transfer framework for expressive volume visualization (VolVis) via neural radiance field (NeRF). The expressiveness of StyleRF-VolVis is upheld by its ability to accurately separate the underlying scene geometry (i.e., content) and color appearance (i.e., style), conveniently modify color, opacity, and lighting of the original rendering while maintaining visual content consistency across the views, and effectively transfer arbitrary styles from reference images to the reconstructed 3D scene. To achieve these, we design a base NeRF model for scene geometry extraction, a palette color network to classify regions of the radiance field for photorealistic editing, and an unrestricted color network to lift the color palette constraint via knowledge distillation for non-photorealistic editing. We demonstrate the superior quality, consistency, and flexibility of StyleRF-VolVis by experimenting with various volume rendering scenes and reference images and comparing StyleRF-VolVis against other image-based (AdaIN), video-based (ReReVST), and NeRF-based (ARF and SNeRF) style rendering solutions.














	[ < g r a p h i c s > < g r a p h i c s >;                     ]




Given a set of multi-view DVR images produced from a volumetric dataset, StyleRF-VolVis transfers arbitrary styles from reference images to the reconstructed 3D scene, synthesizing high-quality novel views with visual content consistency.
  


















figs/figures/pictures/images/./ 


                      
                  
                    
                       




                  











         




 






 














KwInputInput                
KwOutputOutput              

   
  
 








myitemize

 
 
 
 


 










Introduction

StyleRF-VolVis: Style Transfer of Neural Radiance Fields
for Expressive Volume Visualization
    
  Kaiyuan Tang0009-0001-3512-0112 and
  Chaoli Wang0000-0002-0859-3619

    August 5, 2024
============================================================================================



Since the early 1990s, volume visualization (VolVis) has been a central topic in scientific visualization research. One of the most popular techniques for VolVis is direct volume rendering (DVR). It works by casting rays from the image plane to the 3D volume, gathering samples along each ray, mapping them to visual quantities (i.e., colors and opacities) via transfer function (TF) lookups, and compositing the final color for each pixel in the rendering image. Thanks to the astonishing advances in graphics hardware, DVR can be efficiently implemented to achieve interactive framerates and high-quality visualizations, providing superior capability for users to explore volumetric datasets interactively. This practice has been the norm for the past two decades. 

The recent surge of deep learning for scientific visualization research <cit.> has sprouted new opportunities for VolVis. Leveraging the capabilities of generative adversarial networks (GANs), one can train a network to synthesize rendering results under novel viewpoints, TFs, or other parameters, eliminating the need to access the original volumetric data and bypassing the conventional rendering pipeline <cit.>. These seemingly impossible advances could have shocked many researchers just several years ago but are widely understood by the research community nowadays. After all, generative AI has swept across many fields, culminating in its extraordinary power to synthesize novel images and videos from text prompts and level the playing field for the masses.  

Inspired by this fantastic trend, in this paper, we aim to design a deep learning framework to accomplish expressive visualization via style transfer for VolVis. 
Similar to prior work <cit.>, we are supposed to be given a set of 2D images captured from the sample viewpoints and rendered with a fixed, reasonably good, yet unknown TF. Also, we assume no access to the 3D volumetric dataset during training or inference. 
We aspire to develop a new model that achieves expressive visualization by meeting the following goals (G1 to G3) for end users. 

First, they can freely explore the DVR scene in excellent visual quality from previously unseen viewpoints (G1).

Second, they can further flexibly recolor the rendering results photo-realistically by editing the underlying colors, opacities, and lighting effects (G2). 

Third, and most importantly, they can even intuitively modify the rendering results non-photorealistically by transferring styles (e.g., salient brushstrokes) from any image or painting of interest to the DVR scene (G3). 

Realizing G1 is relatively straightforward. 
Unlike GAN-based techniques, state-of-the-art solutions based on neural radiance field (NeRF) can deliver high-resolution, high-quality novel view synthesis results using a set of sample images. 
Nevertheless, significant challenges exist for achieving G2 and G3 due to the considerable gaps between the given 2D images and the 3D scene we aim to reconstruct and edit. 
For G2, the main challenge lies in accurately extracting the contributing color, opacity, and lighting information for faithful downstream edits to ensure visual content consistency. 
For G3, the grand challenge is to move beyond a fixed number of colors decoded from the original rendering images and adapt to abundantly rich color patterns or textures in the reference images. 

We introduce StyleRF-VolVis, achieving style transfer of NeRFs for expressive VolVis. 

For large volumetric data, NeRF-based scene representations are space efficient and thus have implications for altering renderings in a compressive manner. 
At its core, we highlight three key components of StyleRF-VolVis to realize the stated goals (G1 to G3). 

First, we employ a base NeRF model to learn the density representation (i.e., content) of the 3D volumetric data from the collection of 2D training images (G1). 

This component accurately extracts the underlying scene geometry, paving the way for successful subsequent style editing.

Second, we design a new palette color network that classifies regions of the radiance field (RF) and extracts a color palette (i.e., appearance) from multi-view images (G2). 

This component enables photorealistic style editing by modifying the original rendering's color, opacity, and lighting while maintaining visual content consistency across the views.

Third, to remove the color palette constraint, we propose a novel knowledge distillation solution that transfers the color information from the palette color network (i.e., teacher model) to an unrestricted color network (i.e., student model) (G3). 

This component allows users to assimilate a wide spectrum of color patterns and textures from any reference image, supporting expressive visualization of the DVR scene via non-photorealistic style editing. 

Figure <ref> highlights the capability of StyleRF-VolVis on two datasets, each using two styles extracted from different reference images. 
The contributions of our work are:



  * We revisit style transfer for VolVis by presenting an innovative NeRF-based solution that supports expressive photorealistic and non-photorealistic style editing.

  * StyleRF-VolVis represents a significant leap forward for visualization synthesis of volumetric data, advancing the state-of-the-art solutions in quality, consistency, and flexibility. 


  * We show the consistency and flexibility of StyleRF-VolVis on various combinations of DVR scenes and reference images.

  * We compare StyleRF-VolVis with other image-based (AdaIN), video-based (ReReVST), and NeRF-based (ARF and SNeRF) style rendering solutions to demonstrate its superior quality. 








§ RELATED WORK


This section discusses related work on deep learning for VolVis, style transfer, NeRF for stylization, and knowledge distillation.

Deep learning for VolVis.
Over the past few years, deep learning has emerged as a promising solution for improving the DVR process. 

One application is to use deep learning models to replace the traditional DVR pipeline. 
For instance, Berger et al. <cit.> constructed a GAN to synthesize volume-rendered images by investigating the image space of volume rendering under various TFs and viewing parameters. 
Hong et al. <cit.> proposed a GAN to synthesize high-resolution images from volume data under the desired rendering effect without knowing the TF. 
He et al. <cit.> developed InsituNet, a surrogate model that correlates simulation and visualization parameters with visualization outcomes, allowing users to preview visualization results under different simulation settings with a trained model. 
Shi et al. <cit.> built a view-dependent neural-network-latent-based surrogate model that supports producing high-resolution visualization results. 
Han and Wang <cit.> presented CoordNet, a coordinate-based neural network to synthesize rendering images under novel viewpoints given a set of DVR images for training. 

Another application employs a scene representation network (SRN) to represent volumetric data, minimizing disk storage requirements and enabling interactive neural rendering without direct access to volume data. 
For example, Weiss et al. <cit.> designed a dense-grid encoding method fV-SRN that directly renders from compressed representation with no additional memory for storing volume data during rendering. 
Wu et al. <cit.> leveraged multiresolution hash grid encoding and achieved fast volume encoding and real-time rendering. 
Wurster et al. <cit.> developed APMGSRN, an adaptively placed multi-grid SRN with a domain decomposition training approach for efficient VolVis. 

Other deep learning works in the context of VolVis focus on data or visualization generation <cit.>, compression <cit.>, translation <cit.>, and completion <cit.>. 

Our work represents the first step in applying style transfer techniques to volume rendering results using a NeRF model. Compared with existing works <cit.>, StyleRF-VolVis takes a smaller set of DVR images for training and infers higher-quality results, maintaining content consistency across different viewpoints and supporting flexible style transfer. 


Style transfer.
Ever since Gatys et al. <cit.> demonstrated the capability of convolutional neural networks (CNN) in applying diverse artistic styles to natural images, neural style transfer <cit.> has emerged as a trending topic in both academia and industry. Numerous techniques have been developed to improve or extend the original algorithm.

Johnson et al. <cit.> introduced a feed-forward network to solve the slow optimization problem and achieved real-time style transfer. 
Huang et al. <cit.> further maintained speed comparable to <cit.> without sacrificing the flexibility of transferring inputs to arbitrary new styles. This was realized using an adaptive instance normalization layer that aligns the statistics information of the content and style features. 
Ruder et al. <cit.> extended style transfer to video sequences by computing optical flow to achieve consistent style transfer across frames. 
Wang et al. <cit.> presented ReReVST that relaxes the objective function of style loss to make the transfer more robust to motions in content video. 


In VolVis, Bruckner and Gröller <cit.> introduced a style TF using the lit sphere, which assigns the optical properties of voxel values with rendering styles instead of simple colors and opacities. 
However, extracting or designing the lit sphere requires artist involvement, and the rendering relies on a traditional pipeline. Our StyleRF-VolVis adopts an advanced NeRF representation, supporting end-to-end style transfer and rendering without accessing the original volume. 

NeRF for stylization.
Since the groundbreaking work of Mildenhall et al. <cit.> in 2020, NeRF has been widely used for novel view synthesis. 

We refer readers to recent survey papers <cit.> to follow the roadmap of NeRF-based applications.
While extensive studies <cit.> emphasize the quality or speed enhancement of NeRF, there is also a growing body of work focusing on editing a base NeRF to perform 3D style transfer of a scene. 
Generally, editing NeRF for style transfer can be classified into photorealistic and non-photorealistic.

One main task for photorealistic editing is recoloring the scene. 
For example, Kuang et al. <cit.> and Gong et al. <cit.> utilized optimizable base colors in a palette to fit the scene and modify the base colors to recolor the scene during inference.
Lee et al. <cit.> modified the essential weights in the color multilayer perceptron (MLP) of a trained NeRF to perform local recoloring. 

Unlike photorealistic editing, non-photorealistic editing transfers more abstract information, such as textures or brushstrokes of an artistic work. 
For instance, Chiang et al. <cit.> utilized a hypernetwork to transfer style features into the NeRF representation. 
Huang et al. <cit.> proposed a mutual learning strategy to integrate 2D stylization with NeRF geometry consistency. 
Zhang et al. <cit.> designed a nearest neighbor-based loss to stylize a trained NeRF, which performs better stylization quality compared with standard Gram matrix-based loss. 
Liu et al. <cit.> transformed the grid features within RFs to match a reference style. Even though the quality is less impressive than <cit.>, it supports zero-shot transfer to an unseen reference image.

StyleRF-VolVis differs from all these works in that it supports photorealistic and non-photorealistic editing in a single framework. We will show that conducting photorealistic editing before non-photorealistic editing yields better quality and flexibility for stylization results.





Knowledge distillation.
Hinton et al. <cit.> introduced knowledge distillation (KD), which optimizes a small network to match the predictions of a large model. KD has been used in model compression and optimization <cit.>. 
In the NeRF space, Reiser et al. <cit.> leveraged a large global pre-trained NeRF to speed up the training of multiple small local NeRFs. 
Barron et al. <cit.> improved the rendering quality with an online distillation strategy. 
Wang et al. <cit.> designed R2L to learn the light fields of a pre-trained NeRF model effectively. 
Fang et al. <cit.> proposed progressive volume distillation, which provides a systematic KD method that allows the distillation between explicit and implicit NeRF architectures. 

Instead of applying KD to optimize a small network for model compression or optimization purposes, we utilize KD to transfer the knowledge of a photo-realistically edited RF to improve the quality of later non-photorealistic editing.









§ STYLERF-VOLVIS





The objective of StyleRF-VolVis is to edit the style (i.e., appearance) of the DVR scene represented by a NeRF model without losing the original content (i.e., geometry) information. The style can be photorealistic (such as adjusting the original rendering's color, opacity, and lighting attributes) or non-photorealistic (such as altering the texture to resemble the brushstroke of an artistic image). 
Distinct from conventional 3D neural stylization methods, our StyleRF-VolVis aims to unify two types of style editing into a single framework. 
Additionally, from a visualization perspective, users design TFs to classify the regions of interest in the DVR process. 
These TFs map distinct colors and opacities to each region users specify, highlighting relevant data ranges through visual mapping. 
Such visual content is critical in obtaining helpful visualization. Our method must keep the original visual content consistent across the views to ensure the stylized visualization remains meaningful, facilitating accurate interpretation and analysis.





Figure <ref> illustrates the three-stage workflow of StyleRF-VolVis: 
(a) a base NeRF model for optimizing the density representation of the scene, 

(b) a palette color network (PCN) for supporting photorealistic style editing (PSE) and maintaining visual content consistency during style editing, and 
(c) an unrestricted color network (UCN) for enabling non-photorealistic style editing (NPSE). 
Accordingly, our method is structured into three optimization stages, each dedicated to training the parameters of one network while others are frozen for stability.

The first stage focuses on training a base NeRF model to reconstruct the scene geometry for subsequent style editing. 
The parameters of the trained base NeRF are then frozen to ensure that the following style editing does not affect the underlying scene geometry.

The second stage optimizes the PCN. 
Similar to TFs that differentiate volumetric regions with user-defined colors and opacities, the goal here is to classify regions of the RF with 

a color palette extracted from multi-view images. 
To this end, the PCN is optimized to fit input images with a weighted linear combination of extracted palette colors, given the density representation obtained from the previous stage.


By learning to represent the colors of the RF, our model achieves PSE, including modifying the original rendering's color, opacity, and lighting.


The number of palette colors inherently restricts the network's ability to utilize more diverse color patterns for NPSE, where each style often consists of multiple colors. 

To address this issue, 
in the third stage, we start with stylizing the palette colors to match the target style colors. 
Leveraging KD, we then transfer the color information from the PCN (i.e., teacher model) to a UCN (i.e., student model), representing the colors of the RF without the color palette constraint. 
In addition, we can apply different styles to different visual regions extracted from the PCN.  
As a result, we stylize the visualization scene with NPSE while maintaining the original visual content.

The rest of Section <ref> is structured as follows. In Section <ref>, we provide the preliminary knowledge of NeRF representation <cit.> and how we optimize the base NeRF model. In Sections <ref> and <ref>, we discuss how we adapt NeRF's palette-based recoloring strategy <cit.> to fit PCN and achieve PSE. The utilization of KD <cit.> to extract UCN was described in Section <ref>, followed by the NPSE details that combine stylization loss <cit.> and unsupervised segmentation techniques <cit.> in Section <ref>. Finally, we brief our interactive interface design in Section <ref>.





 §.§ Base NeRF Model
 


StyleRF-VolVis is built upon the scene representation from a base NeRF model. 
In general, a NeRF model <cit.> represents the scene with two neural functions: a density function σ(x) that maps any 3D position x to a density value σ and a color function c(x, d) that outputs RGB color c given an arbitrary 3D position x and viewing direction d. 
NeRF samples rays from the camera origin o to the rendering pixels for each training camera view d. For one sample ray r=(o,d), if we sample M points along ray r with 3D positions x_1… M at depths t_1… M. The predicted pixel color ĉ (r) of ray r is computed as

    ĉ(r)=∑^M_i=1α_i(1-ω_i)c(x_i, d),

where ω_i = exp(-(t_i-t_i-1)σ(x_i)) represents the transmittance along the ray between 
x_i and x_i-1. 

α_i=∏^i-1_j=1ω_i denotes the attenuation of the ray from its origin o to 
x_i.

We select the advanced architecture from Instant-NGP <cit.> to optimize the RF, ensuring fast convergence and efficient rendering. Specifically, Instant-NGP comprises a multiresolution hash-grid encoding and a tiny MLP for efficient optimization.
During the optimization of the base NeRF, we optimize with a loss ℒ_ defined as

    ℒ_ = ||c(r) - ĉ(r)||^2_2,

where c(r) and ĉ(r) are the ground-truth (GT) and predicted pixel colors corresponding to ray r.
After optimizing the base NeRF, we freeze its model parameters in the subsequent stages to avoid style editing influencing the scene geometry. Note that the colors of the base NeRF will not be used in rendering the stylized RF; we employ the PCN for PSE and the UCN for NPSE. 









 §.§ Palette Color Network (PCN)



After training the base NeRF model for accurate density representation, we optimize a PCN that represents the scene appearance with 
colors from a palette. 

The trained PCN can classify any 3D position within the RF with its palette color. 
The classification of the PCN preserves region classification from the user-defined TF, ensuring visual content consistency of the downstream style editing.
Figure <ref> shows the PCN architecture. 
The network is composed of two branches: 
one branch uses the lighting MLP to predict the view-dependent specular color c_s and 
the other branch uses the palette MLP to output the view-independent diffuse color c_d. 

For the specular color branch, we use a structure similar to the Instant-NGP's color function. The lighting MLP also ingests geometry features output from the density MLP and spherical harmonics (SH) encoded view direction as input. The only difference is that the output of the lighting MLP is a grayscale color instead of the original RGB color. Before optimization, we leverage the hidden parameters of the base NeRF's color function to initialize the shallow layers of the palette MLP to improve the prediction of c_s at the early stage.





For the diffuse color branch, we design a palette MLP that captures the weights ω of palette colors P for any points within the RF.
This way, we can use ω to classify points in the RF according to their primary colors, preserving visual content consistency across different views during style editing.

Similar to the recent NeRF's palette-based recoloring strategy <cit.>, we apply the RGB convex hull method <cit.> to extract colors P_ from training images. However, unlike previous palette recoloring methods that directly employ P_ to initialize the palette, we further refine P_ as it usually contains many similar colors that may significantly hinder the classification. We remove similar colors when the distance between two colors is below a certain threshold. This distance is measured by the hue component of colors in the hue-saturation-brightness (HSB) space. The saturation and brightness components of palette colors are not considered in measuring the distance as they can be easily optimized during training. Figure <ref> gives an example that compares classification and rendering results without and with palette refinement. Refer to the appendix for more details about the refinement algorithm.

Once the palette colors are initialized and the number of palette colors N_P is determined, we construct and optimize the PCN. During the optimization, the density MLP and hash-grid encoder E_σ learned in the first stage are fixed for invariant density representation. The palette MLP takes features output from the palette hash-grid encoder E_p as input and outputs three values: 
an intensity value I shared by all palette colors to counteract the range shift caused by the RGB convex hull color normalization, 
a set of weights ω that indicate the contribution of each palette color to c_d, and 
a color offset vector δ to enhance the expressiveness of palette colors. 
Given the refined palette colors P and specular color c_s, the output color c associated with position x and view direction d is computed as

    c(x,d) = c_s(x,d)+I(x)∑_i=1^N_Pω_i(x)(P_i+δ_i(x)),

where I(x) is the intensity value at position x, ω_i(x) and δ_i(x) denote the weight and color offset for palette color P_i at x, respectively.
We then predict pixel color ĉ(r) using Equation <ref> for c(x,d), and optimize network parameters and palette colors with loss ℒ_ defined as

    ℒ_=||c(r) - ĉ(r)||^2_2 + λ_δ∑_i=1^M∑_j=1^N_P||δ_j(x_i)||^2_2.

The first term of ℒ_ is the MSE loss between the GT pixel color c(r) and predicted pixel color ĉ(r). 
The second term is the offset regularization loss for sample points along ray r, aiming to suppress the magnitudes of color offsets and avoid significant palette color shiftings. We set λ_δ = 0.1 to control the regularization strength. 





























 §.§ Photorealistic Style Editing (PSE)



When inferring the PCN, the output palette weights ω(x) can classify any 3D position within the RF according to its palette color. We then tune the network values to support PSE. Interactive PSE is achieved using Instant-NGP's fast inference ability.

We use the HSB space for recoloring following <cit.>. Given the target palette colors P', we compute the difference ΔP between the original palette colors P and P' in the HSB space. Δ P_i are then added 
to P_i+δ_i(x)

in Equation <ref>. 
To achieve opacity or lighting editing, we multiply σ(x_P_i) or c_s(x_P_i, d) by a scalar value, where x_P_i denotes a point position receiving its primary color contribution from P_i. 
















 §.§ Unrestricted Color Network (UCN)



Although the PCN can represent the colors of the RF with PSE, it prohibits using diverse colors in NPSE. 
To address this issue, we utilize KD to optimize a UCN (student model) that produces similar diffuse colors within the PCN (teacher model) with no color palette constraint. 

Before distillation, we initially extract the average palette colors P̅ from each reference style to replace the original palette colors P.
This color transfer step ensures the distilled student network can match with styles, reducing optimization effort in NPSE. 
Our UCN comprises one hash-grid encoder E_u following an unrestricted MLP that outputs color c_u. 
During distillation, we first randomly sample camera views from the scene. Then we optimize E_u with volume-aligned loss ℒ_ adopted from <cit.> for each view. ℒ_ is defined as

    ℒ_ = ||E_p(x) - E_u(x)||^2_2,

where E_p(x) and E_u(x) are the output features from the palette and unrestricted encoders for each sample point x. We optimize ℒ_ for 150 iterations to initialize E_u(x) to accelerate the subsequent distillation. For the following 500 iterations, we optimize both E_u(x) and unrestricted MLP with color loss ℒ_, which is defined as

    ℒ_ = ||c̅_d(x) - c_u(x)||^2_2,

where c̅_d(x) and c_u(x) are the diffuse colors output from the PCN with the average palette colors P̅ and the UCN at x.  





 §.§ Non-Photorealistic Style Editing (NPSE)






Given one or multiple reference images, we first extract N_P desired reference styles S={S_1, S_2, ⋯, S_N_P} for N_P regions of the RF. 
When the number of reference images is less than N_P, or users are only interested in local patterns of one image, 
we can leverage unsupervised segmentation techniques such as the segment anything model (SAM) <cit.> to automatically or manually (with point prompt selection) divide a reference image into localized style regions for flexible stylization.

Figure <ref> shows our training process for non-photorealistic style optimization. We use the optimizable unrestricted color c_u(x) (Section <ref>) and frozen density σ(x) of the base NeRF (Section <ref>) to construct a new RF. 

During training, we first compute N_P rendering images R={R_1, R_2, ⋯, R_N_P} from the new RF for each camera view. 
For one sample point x, it only contributes to R_i if its palette weight ω_i(x) is greater than other palette weights.
Once we have N_P renderings R and styles S, a pre-trained VGG-16 model <cit.> computes a stylization loss based on a user-specified or randomly assigned style mapping. Specifically, we use the nearest neighbor feature matching (NNFM) loss <cit.> for each rendering and style, which is formulated as

    ℒ_(R,S)=1/(N_FN_P)∑_i=1^N_P∑_f_j∈ϕ(R_i)min_f_k∈ϕ(S_i)d(f_j,f_k),

where ϕ(R_i) and ϕ(S_i) are the rendering and style feature vectors extracted from the VGG-16 model ϕ. N_F is the number of feature vectors for ϕ(R_i), and d is the cosine distance between feature vectors f_j and f_k.





Unlike previous NeRF-based non-photorealistic stylization methods <cit.>, our style transfer process involves three unique designs for the VolVis scenario. 
First, different from traditional NeRF-based stylization that directly discards the view-dependent color to maintain visual content consistency across the views, we optimize c_u(x) (i.e., diffuse color) during training and output c_u(x)+c_s(x,d) (i.e., diffuse and specular colors) during inference. The rationale behind this design is to prevent the view-dependent color from influencing visual content consistency and preserve the specular lighting effect in the final output. 
Second, when rendering R in training, we observe that varying background colors could impact the stylization results (refer to an example shown in Figure <ref>). This phenomenon is caused by the inherent transparency in the DVR scene, which is not often present in natural scenes. Consequently, different background colors could lead to distinct renderings using the identical network, potentially incurring unsatisfactory stylization. 

To address this issue, we extract the luminance value from each style S_i as the corresponding background color for each rendering R_i.
Third, we omit the content loss between the original and stylized images due to suboptimal stylization outcomes.













 §.§ Interactive interface


Figure <ref> shows our visual interface for users to conduct PSE and NPSE for a given DVR scene. 
For PSE, users select a palette ID and interactively modify the corresponding scene region's color, opacity, and lighting. 
For NPSE, users first specify the target style for each scene region from single or multiple reference images. 
They then start optimizating UCN. 
After NPSE, users can still change the opacity and lighting of the stylized DVR scene but cannot modify the color as the stylized scene is represented with UCN. 
Refer to the accompanying video for the recorded interaction with the interface.














§ RESULTS AND DISCUSSION






 §.§ Datasets, Training, Baselines, and Metrics


Datasets and network training.
To show the quality, consistency, and flexibility of StyleRF-VolVis, we evaluate it using the datasets listed in Table <ref>. 
The visible ranges are the opacity bumps in the TF specification corresponding to distinct visual contents for generating the original rendering. 
All reference images used are copyright-free, coming from WikiArt <cit.> and Pexels <cit.>. 
We implemented StyleRF-VolVis using PyTorch and ran experiments on an NVIDIA RTX A4000 graphics card with 16 GB memory. 
We trained the base NeRF model with 30,000 iterations and PCN with 2,500 iterations using 92 DVR images with cameras evenly placed along a sphere enclosing the volume. 
The KD of UCN and NPSE does not need GT images. 
For KD, we optimized UCN with 624 iterations, where each iteration trains on one of the randomly sampled 312 camera views. 
For NPSE, we optimized UCN using the NNFM loss with 210 iterations, where each iteration trains on one of the uniformly sampled 42 camera views. 
For all three stages (base NeRF, PCN, and UCN), we applied the Adam optimizer with a learning rate of 0.01 and β_1=0.9, β_2=0.999 for training and set the batch size to 4,096 rays.
For the base NeRF model, PCN, and UCN during KD, we decayed the learning rate exponentially for every iteration until it reached 0. 
During NPSE, we fixed the learning rate and followed the setting of ARF <cit.>. 
We used the conv3 block of VGG-16 to extract features for computing the NNFM loss. 
Deferred backpropagation was employed to update UCN's parameters for the entire image under limited GPU memory. 
During inference, we generated 181 stylized images with novel views, with cameras evenly placed along a trajectory to capture the full 360-degree view. 


Baselines.
We compare four baseline methods for NPSE:



	
  * ARF <cit.> is a NeRF-based method that trains a NeRF model to represent the scene and then updates the pre-trained NeRF parameters with NNFM loss to generate stylized novel view images. 
	
  * SNeRF <cit.> is a NeRF-based method that first constructs the scene with a NeRF model and then alternatively finetunes the parameters of the pre-trained NeRF and an additional image stylization module to achieve stylization. 
	
	
  * ReReVST <cit.> is a video-based stylization method. It first performs novel view synthesis with a NeRF model, then arranges the novel view images as a video sequence, and finally performs video stylization to yield the result. 
	
  * AdaIN <cit.> is an image-based stylization method that optimizes a NeRF model for generating novel view images of the scene and then applies stylization to the NeRF synthesized images.




Note that we did not compare StyleRF-VolVis with the more recent StyleRF <cit.> work as it mainly focuses on the ability of zero-shot transfer to unseen reference images. 

StyleRF trains on all reference images from the WikiArt dataset to ensure better generalization for unseen reference images. However, this could lead to inferior stylization results compared to optimization methods like ARF and SNeRF.  
Existing visualization generation methods in scientific visualization <cit.> were also excluded because they were not designed for style transfer. 

Evalutation metrics.
Since 3D style transfer is relatively novel and scarcely explored, limited metrics are available for quantitatively assessing stylization quality. 
Consequently, our evaluation focuses on cross-view consistency of stylization results. 
Specifically, we generated testing videos in which each frame represents a rendered image from one of the 181 novel viewpoints of our stylized scene. Then, we warped one view to the other using softmax splatting <cit.> according to the optical flow estimated with RAFT <cit.>. Finally, we computed the MSE and LPIPS <cit.> scores to measure the cross-view consistency. 
Similar to <cit.>, we calculated the long-range consistency for faraway views (with an interval of 10 among the 181 inference views) and short-range consistency for adjacent views, respectively.





 §.§ Qualitative Comparison


We compare StyleRF-VolVis against baseline methods using DVR scenes of vortex, supernova, combustion, and five jets. Each scene showcases two stylization results using distinct reference images. Figure <ref> shows the qualitative comparison. Unlike all baseline methods, StyleRF-VolVis supports customized style selection and mapping to distinct visual contents, which we highlight in the DVR scene.

Overall, StyleRF-VolVis produces clearer and more consistent stylization results than other methods. 
In contrast to video- and image-based stylization methods (ReReVST and AdaIN), NeRF-based stylization methods (StyleRF-VolVis, ARF, and SNeRF) preserve better geometry consistency and are capable of separating foreground objects from the background during stylization. 
Among NeRF-based methods, SNeRF captures the overall style of the reference image but cannot assign distinct styles to different visual regions of the original scene. Thus, SNeRF's stylization is blurry and mixed, making it hard to identify the difference between individual visual contents in the original DVR scene.

For the vortex dataset, ARF shows better stylization than SNeRF in preserving visual content distinction. 

However, the uncontrollable ARF style selection process could let the model choose a negligible local style that does not match the overall style (e.g., the WikiArt-1 case) or select one style for multiple visual regions (e.g., the WikiArt-8 case). Both drawbacks could lead to poor stylization results.
In addition, in the stylization process, ARF and SNeRF discard view directions in their input to ensure cross-view stylization consistency. Consequently, they cannot preserve the view-dependent lighting of the DVR scene.

Unlike ARF and SNeRF, StyleRF-VolVis utilizes the lighting MLP to preserve lighting information of visual content while ensuring cross-view consistency during NPSE by removing view direction input in UCN optimization. 
Moreover, StyleRF-VolVis allows explicit assigning of different styles to different DVR regions. Via color transfer, we ensure that the region color always aligns with the overall style color, avoiding the disadvantage of NNFM loss, which focuses on negligible local style. 































 §.§ Quantitative Comparison


Training and inference time. 
We report the training and inference time for all NeRF-based methods in Table <ref>.
The training of StyleRF-VolVis is slower than ARF but faster than SNeRF.
Unlike ARF, StyleRF-VolVis requires additional steps to optimize PCN and apply KD to UCN before NPSE. 
Moreover, optimizing on a single camera view during NPSE requires rendering different regions independently. 
This process maps a style to a region, slowing the stylization process. 
Although the alternative training process of SNeRF is relatively straightforward, it demands significant time to train the image stylization module, leading to the longest training time. 
For inference, StyleRF-VolVis needs additional feedforward steps of the lighting MLP to provide lighting information, which takes longer than ARF or SNeRF. 







Short- and long-range consistency.
In Table <ref>, we compare short- and long-range cross-view consistency for different methods. 
We use MSE and LPIPS as the metrics. 
For each metric, we report the average value over eight stylization cases (WikiArt-1 to WikiArt-8) shown in Figure <ref>. 
StyleRF-VolVis achieves the best cross-view consistency, while AdaIN performs the worst as this image-based stylization method treats each image independently. 
ReReVST leads to comparable or even better MSEs than the NeRF-based methods because it produces stylization with relatively uniform colors, resulting in smaller pixel-wise errors. 
However, under the image-level LPIPS metric, the consistency of ReReVST significantly lags behind the NeRF-based methods.





Voting results from a user study.
As the consistency metrics do not necessarily reflect the perceived style transfer quality, we conducted a user study to measure user preference for different stylizations following the University's IRB protocol.

Per Figure <ref>, we included four datasets, each with two reference images, and considered three leading methods (ARF, SNeRF, and StyleRF-VolVis) for pairwise comparison. This leads to 24 (4×2×3) image pairs organized into eight (4×2) groups. 

We recruited 14 students from a visualization class of undergraduate, Master's, and Ph.D. students in computer science and engineering, aerospace and mechanical engineering, and psychology majors. 

The participants were briefed on the evaluation criteria before proceeding to the study. 
A full-screen display shows each pair. At the top of each display, the original DVR and reference images (with extracted style regions highlighted in different color boundaries) are shown. At the bottom, the stylized images of two methods, randomly placed on the left and right sides and labeled `A' and `B,' are presented. 
The participants were asked to decide which one (`A' or `B') achieved the better stylization outcome, and no tie was allowed.

We asked the participants to take their time, as we did not record how much time they spent on each pair, each group, or the entire study. During the evaluation, they could go back and forth to update their votes. 
We advised them that many factors should be considered, including the overall impression, content preservation, style application, and visuals (color, opacity, and lighting). 
They could decide how to weigh them, and their criteria should be consistent throughout the study.

The entire study was completed in the classroom within 10 minutes. 

The voting results are shown in Table <ref>. 
We can see that StyleRF-VolVis wins for all stylization cases except WikiArt-1, WikiArt-3, and WikiArt-8. This may be attributed to the fact that some participants prefer 
the stylization result of ARF that provides a detailed local pattern for the WikiArt-8 case or 
the stylization results of SNeRF, which contain a more global pattern in WikiArt-1 (i.e., the dark tone and contrasting colors of The Starry Night) and WikiArt-3 cases. However, StyleRF-VolVis wins for all other cases and gains the most overall preference. The cross-view consistency of StyleRF-VolVis is also better than ARF and SNeRF, as shown in Table <ref>.









 §.§ Flexibility of NPSE



Our NPSE strategy allows users to define which style should be applied to which part of a DVR scene (analogous to using the TF), which provides more flexibility than other NPSE methods. Specifically, users can apply styles obtained from one reference image for multiple stylizations, leverage styles extracted from multiple reference images for one stylization, or only stylize one part of the DVR scene. All these additions are the unique features of StyleRF-VolVis.

One reference image for multiple stylizations.
Unlike conventional NeRF-based stylization methods, which only produce one stylized scene from a single reference image, StyleRF-VolVis can generate various stylized outcomes with only one reference image.
Thanks to the advanced segmentation of PCN and SAM for the RF and reference image, we can extract multiple distinct styles and apply them to different scene regions.
By pairing styles and regions in various combinations, StyleRF-VolVis achieves controllable and diverse stylization outcomes.
Figure <ref> shows multiple stylization results of the solar plume DVR scene using the same reference image.
By comparing Pair-1 and Pair-2, we can see that when the style of one region remains unchanged, altering the style of another region keeps the unchanged style region intact. 
Further comparison of Pair-2 and Pair-3 reveals that applying the same style to different regions consistently transfers the expected texture.
These favorable features allow StyleRF-VolVis to offer users flexible stylization.
 




Multiple reference images for one stylization.
Previous approaches like StyleRF <cit.> suggest leveraging extra supervision from large models pre-trained on extensive image datasets for segmenting scene elements to support style transfer from multiple reference images to a single scene. 

However, such a strategy cannot be directly applied to VolVis scenes. 
One reason lies in the inherent difference between DVR and natural scenes. 
DVR images exhibit more complex geometric relationships, with different visual contents often nested in layers with varied transparencies, rendering 2D segmentation methods futile. 
Another reason is that large models are typically pre-trained on natural images. 
Adapting such models to process DVR images would require extensive DVR images and substantial hardware resources for fine-tuning, a demand that is impractical for user-oriented VolVis scene stylization.

Our model can segment various regions via PCN, even though separate GT rendering results for different regions are missing during optimization and additional supervision from a 2D segmentation method is lacking. 
As illustrated in Figure <ref>, StyleRF-VolVis achieves stylized results for different regions using styles from multiple reference images. 
With a collection of reference images, StyleRF-VolVis could create vast combinations of different stylization results for one DVR scene.





Partial stylization.
In some scenarios, users may wish to stylize only certain regions of the scene while keeping others unchanged.
Unlike previous NeRF-based stylization methods, StyleRF-VolVis achieves such partial stylization by combining the color representations of PCN and UCN. 
After completing the NPSE of each region, PCN can still utilize the original palette colors to represent the DVR scene. 

Users can thus specify which regions should maintain the NPSE texture and which should keep the original appearance. 
During rendering, regions that preserve the original scene's appearance utilize colors represented by PCN, while those stylized regions are processed through UCN.
Figure <ref> shows an example of partial stylization, providing extra freedom for users to achieve desirable results.









 §.§ PCN Classification and PSE


As discussed in Section <ref>, the PCN classification outcomes are crucial for style editing.
Visual contents in a DVR scene are often nested with each other. 
However, as demonstrated in Figure <ref>, even for a complex scene like the earthquake, PCN can still produce region classification closely resembling GT, even though only DVR images are used for training.
Although the colors and lighting of PCN classification results are not perfect, the boundary of each region is clear, and even fine details maintain consistency with GT. 
This ensures precise PSE and NPSE of each region in subsequent stages.

In Figure <ref>, we showcase the PSE outcomes for various DVR scenes. With the assistance of PCN classification, we can perform a distinct PSE in each region. Such flexibility further assists users in achieving desired stylization results.









 §.§ Ablation Study


Benefit of PSE to NPSE.
At the beginning of UCN optimization, we perform the color transfer 

in PCN (teacher network) to ensure that the color representation of UCN (student network) matches the selected styles before NPSE. 
This PSE step is essential for speeding up the convergence and improving the stylization quality of UCN during NPSE optimization. 
Figure <ref> compares the style transfer process without and with color transfer.
Comparing the stylization results of 42 and 168 iterations, we can see that NPSE without color transfer suffers from inaccurate style color matching at an early stage and shows unclear stylization (i.e., black borders, see arrows) at 168 iterations. In contrast, NPSE with color transfer matches the overall style color well at the early stage and reveals well-stylized texture at 168 iterations. 





View-dependent lighting in NPSE. 
Recent NeRF-based stylization methods <cit.> discard view directions in the input to ensure cross-view stylization consistency. 
However, they ignore that view-dependent lighting of the original scene is essential for maintaining consistency between the original content and the stylized scene.
In contrast, StyleRF-VolVis utilizes the additional lighting MLP optimized in the PCN training stage to preserve the DVR lighting during NPSE. 
In Figure <ref>, we show examples of vortex and supernova datasets to compare the effect of DVR lighting on NPSE outcomes. 
The results show that the stylized scene with lighting is more consistent with the original DVR scene than that without. 









 §.§ Limitations


Although StyleRF-VolVis achieves flexible, high-quality stylization of DVR scenes, it has the following limitations. 

First, given an initial TF, NeRF cannot represent the value ranges where opacity equals zero. Consequently, StyleRF-VolVis cannot perform PSE or NPSE for regions with zero opacity in the initial TF.  

Second, StyleRF-VolVis supports interactive PSE but not interactive NPSE.

Third, even though users can adjust the lighting intensity for the DVR scene, they cannot modify the lighting direction because no normal information is available.

Finally, 

if the colors among different regions within the input DVR scene are similar, color refinement may not separate these regions correctly. In such cases, users may need to manually adjust the number of palette colors and their RGB values to obtain the desired results.






§ CONCLUSIONS AND FUTURE WORK


We have presented StyleRF-VolVis, the first work in VolVis that targets style transfer in the NeRF space. 
The crux of our approach lies in the accurate extraction of content and appearance information separately from the given DVR scene and the bridging between photorealistic and non-photorealistic style editing via knowledge distillation. 
With these innovations, StyleRF-VolVis achieves high-quality, consistent, and flexible 3D style transfer outcomes with novel view synthesis. 
The efficacy of StyleRF-VolVis is demonstrated with various combinations of DVR scenes and reference images. 
Moreover, we compare StyleRF-VolVis against other image-based (AdaIN), video-based (ReReVST), and NeRF-based (ARF and SNeRF) solutions via objective and subjective evaluation to showcase its superior quality performance. 

In the future, we will extend StyleRF-VolVis to handle dynamic DVR scenes produced from time-varying datasets. 
The challenge is maintaining a consistent appearance over timesteps to achieve temporally coherent stylization. 
We will also explore StyleRF-VolVis for multivariate or ensemble VolVis, where different variables or ensembles could be mapped to visually distinct styles beyond colors for better differentiation. 
It remains to be seen what the appropriate number of styles and their mixing should be to leverage the human's visual capacity best while maintaining observation clarity.

To ease the difficulty for non-professionals using StyleRF-VolVis, we will explore integrating natural language interaction into the current graphical user interface and broaden the selection of reference images from the WikiArt collection to images created by generative AI. 

The success of StyleRF-VolVis will unfold exciting opportunities for VolVis beyond the originated scientific domain. We envision that such a solution will enable citizen science by fusing diverse disciplines, such as science and art, for the general public's exploration, understanding, and appreciation, which we would like to pursue. 





This research was supported in part by the U.S. National Science Foundation through grants IIS-1955395, IIS-2101696, OAC-2104158, and IIS-2401144, and the U.S. Department of Energy through grant DE-SC0023145. The authors would like to thank the anonymous reviewers for their insightful comments.



abbrv-doi-hyperref











§ APPENDIX








§ IMPLEMENTATION DETAILS


Palette refinement algorithm.
When initializing palette colors P before PCN training, we refine P_ extracted from the RGB convex hull method

to eliminate similar colors within the palette. 

Specifically, for any two colors in P_, if they are similar (i.e., their L1 distance of hue value in the HSB space is below a threshold T_h), we remove one of them. 
Furthermore, there is a small chance that P_ may contain gray colors corresponding to the light color in the DVR scene. 
Such gray colors are unnecessary in P as the lighting MLP has represented lighting components. 
Therefore, we remove any color in P_ if its brightness value in the HSB space is less than a threshold T_b. 
For normalized color component values in the HSB space, we empirically set T_h=0.1 and T_b=0.2 to obtain refined P.









































Luminance background.
Before NPSE, we calculate each style's luminance value L to compute the NNFM loss using VGG-16.
Following the calculation steps given in <cit.>, for each average RGB color of the selected style, we normalize each component, such as R, and convert it into the linear-scale counterpart R_

    R_ =  R^2.2.

L of the selected style can be computed as

    L = 0.2126× R_ + 0.7152× G_ + 0.0722× B_,

where the RGB component coefficients reflect the average spectral sensitivity of lighting perceived by humans. 
We use L as the corresponding background color for each style rendering. 
This way, the stylization of StyleRF-VolVis can better match the overall brightness of the selected style.










§ ADDITIONAL RESULTS


PSE after NPSE.
After NPSE, users can still apply further PSE to the stylized scene. 
When doing so, one limitation is that if a region of the NPSE scene utilizes the UCN to represent the color term, users cannot apply a color PSE to the region as the PCN does not represent the color.
However, users can modify the opacity and lighting of each region without restriction.
Figure <ref> shows examples of applying various opacity or lighting PSEs to the scene after NPSE.













Choice of λ_δ.
When optimizing the PCN to avoid palette color shiftings, we include an offset regularization loss and use λ_δ to control the regularization strength. 
We conduct an ablation study to investigate the effect of λ_δ on PCN performance.
After optimizing the PCN with different λ_δ, we compare PCN rendering results with GT using PSNR, LPIPS, and SSIM, as shown in Table <ref>. 
Figure <ref> presents the rendering results. 

We can see that the offset regularization loss is essential for PCN training. 
When offset regularization is missing (λ_δ=0), the PCN does not predict correctly, as it tries to focus more on leveraging offsets instead of palette colors to represent the DVR scene.
However, the PCN may leverage more on palette colors instead of offsets to represent colors when λ_δ gets larger, resulting in less accurate scene reconstruction. 
Therefore, we choose λ_δ = 0.1 for a good control of the regularization strength.

Iterative style refinement for NPSE.
For NPSE, StyleRF-VolVis does not support direct control over the stylization process. 
However, users can indirectly achieve their desired stylization by iterative refining the selection in the reference image.
Figure <ref> shows such an example.
Based on the style selection in (a), we apply several negative point prompts for SAM in (b) to exclude undesired styles and retrain the UCN to achieve the desired stylization.





























Comparison with NPR.
Conventional non-photorealistic rendering (NPR) of a DVR scene relies on predefined generation rules to mimic a reference style. 
Due to the complexity of hyperparameter settings, the effectiveness of these methods depends on the choice of hyperparameters to some extent.
Moreover, such an approach limits the stylization results to a single style.
In contrast, StyleRF-VolVis can transfer arbitrary styles according to different reference images within a unified framework, offering higher flexibility and robustness.
In Figure <ref>, we compare the stylization results of StyleRF-VolVis and a conventional NPR method for stipple drawing style <cit.>.
We use the stipple drawing in <cit.> as a reference image and adjust the TF and camera pose to align the viewpoint for comparison.
StyleRF-VolVis matches the overall texture of the reference style more closely.





Comparison with DVR.
In DVR, users can adjust the TF to explore the scene. 
StyleRF-VolVis can also achieve similar objectives through the PSE of PCN. 
However, PSE cannot adjust invisible parts of the scene constructed via training the multiview images. 
Despite this limitation, PCN supports random access to any position within the scene during the rendering's sampling process. 
Compared to DVR, PCN may achieve faster render speed and require smaller memory footprints for large volumes under the same TF.
In Table <ref>, we compare DVR and PCN regarding the average rendering time per image, CPU memory, and GPU memory requirement for the rotstrat dataset with different volume and image resolutions (denoted by experiment IDs). 
We run DVR using the open-source software ParaView with NVIDIA IndeX plugins.
All models converge around four minutes and occupy a storage of 168 MB.
PSNR and LPIPS values reported in Table <ref> and the difference images (with respect to the GT rendering image) shown in Figure <ref> suggest that PCN achieves acceptable accuracy but requires a smaller rendering memory footprint and faster rendering speed compared to DVR. 
Note that the NeRF-based representation is independent of the volume resolution.  
Storing the training images and the network model for large data is more space-efficient than the original volume. 
Therefore, StyleRF-VolVis provides an efficient means for altering renderings of large volumes.




 Paper Title: CARMIL: Context-Aware Regularization on Multiple Instance Learning models for Whole Slide Images.








CARMILT. Nait Saada and V.Di Proietto et al.

CARMIL: Context-Aware Regularization on Multiple Instance Learning models for Whole Slide Images
    Thiziri Nait SaadaThese authors contributed equally to this work Owkin, Inc., New York, NY, USA Mathematical Institute, University of Oxford, Oxford, UK naitsaadat@maths.ox.ac.uk 
 Valentina Di-Proietto[1] [2] valentina.di-proietto@owkin.com 
 Benoit Schmauch[2] benoit.schmauch@owkin.com 

Katharina Von Loga[2] katharina.vonloga@owkin.com 

Lucas Fidon[2]  lucas.fidon@owkin.com
    August, 2024
================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Multiple Instance Learning (MIL) models have proven effective for cancer prognosis from Whole Slide Images. 
However, the original MIL formulation incorrectly assumes the patches of the same image to be independent, leading to a loss of spatial context as information flows through the network. Incorporating contextual knowledge into predictions is particularly important given the inclination for cancerous cells to form clusters and the presence of spatial indicators for tumors. State-of-the-art methods often use attention mechanisms eventually combined with graphs to capture spatial knowledge. In this paper, we take a novel and transversal approach, addressing this issue through the lens of regularization. We propose
Context-Aware Regularization for Multiple Instance Learning (CARMIL), a versatile regularization scheme designed to seamlessly integrate spatial knowledge into any MIL model. Additionally, we present a new and generic metric to quantify the Context-Awareness of any MIL model when applied to Whole Slide Images, resolving a previously unexplored gap in the field.

The efficacy of our framework is evaluated for two survival analysis tasks on glioblastoma (TCGA GBM) and colon cancer data (TCGA COAD). 












§ INTRODUCTION




The digitization of histopathology Whole Slide Images (WSIs) and the development of deep learning methods has lead to promising computational methods for cancer prognosis.


One computational challenge is the large size of WSIs, of the order of 100,000 × 100,000 pixels. Processing images of such size with a deep neural network directly is not possible with the GPUs commonly available. 

Overcoming this problem, previous work proposes to tessellate each WSI into thousands of smaller images called tiles and global survival prediction per slide is obtained in two steps.
The tiles are first embedded into a space of lower dimension using a pre-trained feature extractor model, and a MIL model is trained to predict survival from the set of tiles embeddings of a WSI <cit.>.






One limitation of MIL is the assumption that tiles from the same WSI are independent <cit.>. In particular, MIL models fail to leverage spatial interactions between tiles and their ordering in a WSI. 

In contrast, pathologists take into account the spatial organization of WSIs in their analysis.

To tackle this issue, a correlated variant of MIL has been proposed in <cit.>.

In practice, interactions between neighboring tiles in a WSI can be modeled using a graph of tiles.


Graph Neural Network (GNN) <cit.> and Transformers with local attention <cit.> were proposed to capture correlations between neighboring tiles.

However, as these correlations are parameterized using more complex architectures involving a greater number of parameters, the effectiveness of such mechanisms might be limited by the amount of available training samples. Besides, histopathology datasets usually contain only up to a few hundred WSIs.
By contrast, incorporating spatial correlations between tiles into the model through regularization has been under-explored in computational pathology.




In this paper, we introduce the CARMIL framework to enhance the Context-Awareness of any MIL model, initially agnostic to the spatial context, using Regularization. 

In CARMIL, a spatial encoder and a spatial decoder are added between the feature extractor and the MIL model.

In addition, a Context-Aware Regularization (CAR) loss function is proposed to train the spatial decoder to reconstruct the input graph of tiles.


The proposed spatial encoder aims at distilling the spatial relation between neighboring tiles directly into the tile embeddings so that any MIL model can exploit this information.

To this end, the spatial decoder and the CAR loss are designed to encourage bringing closer the embeddings learnt by the spatial encoder for tiles that are spatially close in the original WSI.





To the best of our knowledge, such regularization strategy has so far not been studied in computational pathology. 

The nearest related work is the topologically-aware regularization for cancer blood cell classification introduced by <cit.>. 


Yet, this regularization is based on the 0th order Betti number, which only captures the connected components of the graph of tiles.

Instead, CARMIL aims to reconstruct the entire graph structure.

Our main contributions are three-fold. 

First, we introduce CARMIL, a Context-Aware Regularization module based on Graph AutoEncoders that we incorporate into the classical MIL framework through regularization.

Second, we propose DeltaCon, a novel metric to quantitatively assess the Context-Awareness of any tiles embedding. This metric allows for a better inspection of how well the embeddings conform with the original spatial arrangement of the tiles within the WSIs.

Lastly, we evaluate CAR on a set of benchmark MIL models for the task of overall survival prediction for glioblastoma and colon cancer. These models are precisely chosen because they are originally context-independent, and we show through extensive quantitative and qualitative ablation studies how CARMIL turns them into Context-Aware models and leads to improved C-index.



 











































§ METHODS





 §.§ Background: classical MIL framework for risk prediction


In this section, we summarize the main steps of the classical MIL pipeline, that we enhance with our Context-Aware Regularization in the next section.





  
Preprocessing. 
Each WSI is segmented to keep only the tissue and remove the background (e.g. using the otsu method).

The tissue parts are then tessellated into tiles of size 224 × 224 pixels taken at 20X magnitude (0.5μ m/pixel) and each WSI is reduced to a set of n tiles of tissue selected at random.

A pre-trained feature extractor is then used to map each tile into a low-dimensional tile feature space of dimension d. This results in a matrix representation X^(i)∈ℝ^n × d for the WSI indexed by i.













  
MIL model.

The MIL model f_MIL is a parametric model trained to aggregate the tile features of a WSI and estimate a risk associated with the overall survival of the patient 

    f_MIL: ℝ^n × dℝ.


The parameters θ_MIL of the MIL model f_MIL are optimized to minimize a survival loss ℒ_MIL, such as the Cox loss. Formally, this corresponds to the optimization problem


    min_θ_MIL1/N∑_i=1^N ℒ_MIL(
        f_MIL(X^(i);θ_MIL),  y^(i)),


where N is the number of training WSIs and y^(i) is the overall survival risk of patient i.

Examples of such MIL models are ABMIL <cit.>, Chowder <cit.>, and AdditiveMIL <cit.>.

It is worth noting that the spatial relationships between tiles are not used by f_MIL, since their ordering is random in X^(i).





 §.§ Context-Aware Regularization (CAR)


We propose a general approach to leverage local spatial context in computational histopathology. 

Our pipeline, which can be applied to any task or MIL model, is depicted in fig. <ref>.



  
Graph construction.
For each WSI, we build a graph G^(i)=(X^(i), A^(i)) where the vertices are the tiles features X^(i)∈ℝ^n × d, as described in the previous section, and A^(i)∈ℝ^n × n is the adjacency matrix computed based on the euclidean distance between the spatial coordinates of the tiles in the original WSI for a given number k of nearest neighbors.



  
Context-Aware Regularization.
To embed the spatial structure of the WSI into the tiles features, we introduce a spatial encoder f_E of parameters θ_E and a spatial decoder f_D of parameters θ_D before the classical MIL model f_MIL of (<ref>) as illustrated in Fig. <ref>

    [t]
        f_Eℝ^n × d×ℝ^n × n   ℝ^n × d_E
    
        (X,A)   ↦ Z,
          [t]
        f_D ℝ^n × d_E   ℝ^n × n
    
        Z    ↦Â.


 
Here, the goal of f_E is to distill the spatial information contained in an input WSI graph G=(X, A) 

into tiles features X, resulting in a new low-dimensional Context-Aware embedding Z ∈ℝ^n × d_E of the WSI. 
To achieve this goal, f_D aims at reconstructing the input adjacency matrix A from Z during training while, concurrently, f_MIL aims at predicting the patient risk from Z.

Formally, this corresponds to adding a Context-Aware Regularization term ℒ_CAR to the training optimization problem of classical MIL (<ref>):


    min_θ_MIL, θ_E, θ_D1/N∑_i=1^N( 
        (1-β) 
        ℒ_MIL(
            f_MIL(Z^(i);θ_MIL),  y^(i))
        +
        βℒ_CAR(
            f_D(Z^(i); θ_D), A^(i)) 
        )

where the same notations as in (<ref>) are used and,

    {[                                                 Z^(i) =                                        f_E(G^(i); θ_E),;                                           ℒ_CAR(Â, A) = 1/n^2∑_p,q(A_pqlog ( Â_pq ) + (1-A_pq) log (1 - Â_pq)). ].


By reformulating the training problem as a joint optimization task, the models' parameters are obtained by simultaneously solving a pair of objectives, which may initially appear orthogonal. 

When β=0, the method boils down to the classical MIL pipeline, whereas β=1 reduces the task to encoding the spatial context only as in a Graph AutoEncoder (GAE) <cit.>.

Note that the spatial decoder f_D is not required at inference.

The spatial encoder f_E and spatial decoder f_D are obtained by stacking up, respectively ℓ_E and ℓ_D, graph convolutional network (GCN) layers:



    GCN((X,A); θ)  A ReLU(X)W^θ,

    where W^θ is a learnable weight matrix and the matrix A may be preprocessed. 

If we denote the composition of a function f by itself t times as f^∘ t, then,


    Z = f_E(G;θ_E) GCN^∘ℓ_E(G;θ_E),     
        U_D(Z) = GCN^∘ℓ_D( (Z,A);θ_D).




Similarly to a GAE, the last layer of the spatial decoder is an inner-product decoder with a sigmoid function σ, that aims at reconstructing the input adjacency matrix A,

    Â= f_D(Z;θ_D) σ( U_D(Z) U_D(Z)^T).


    


















 §.§ DeltaCon for quantitative measure of Context-Awareness


Reporting task-related scores alone is not enough to quantify the amount of Context-Awareness in a tiles embedding space.

To overcome this issue, we propose a novel Context-Awareness metric based on DeltaCon <cit.>. 

Let Ẑ∈ℝ^n× p be tiles descriptors, we denote A(Ẑ)∈ℝ^n× n the adjacency matrix based on the k nearest neighbors for the euclidean distance between the components of Ẑ.

This is different from A, defined in sec. <ref>, that is based on the distance between the spatial coordinates of the tiles.

Ẑ can be any representation of the tiles, here it will either be the features X or the embeddings Z as defined in sec. <ref>.

We propose to use the DeltaCon similarity between the adjacency matrix based on tiles descriptors A(Ẑ) and the adjacency matrix based on the spatial coordinates A for the same WSI as our metric of Context-Awareness.

Since we have directed graphs and the adjacency matrices A and A(Ẑ) are non-symmetric, we propose this extension of the original DeltaCon similarity

    DeltaCon(A(Ẑ), A) = 
    1/1 + ||S(A(Ẑ)) - S(A)||_F,    
        S(A)  (𝕀 + ϵ^2 D - ϵ A)^-1,

where D D_in+ D_out, where D_in, resp. D_out, is the diagonal matrix counting the incoming, resp. outcoming, edges and ||.||_F denotes the Frobenius norm.

Intuitively, S(A)∈ℝ^n× n is a matrix capturing the degrees of similarity between all tiles when one travels in the original graph corresponding to A.

Direct neighbors are the most similar tiles, followed by neighbors of neighbors, and so on.

As a result, DeltaCon(A(Ẑ), A) is close to 1 when the neighbors in the tiles descriptors space and the spatial coordinates are almost the same and it smoothly decreases towards 0 when those neighbors become dissimilar between the tiles representations and the spatial coordinates.



















§ IMPLEMENTATION DETAILS



 §.§ Datasets for survival prediction using whole slide images




  
Glioblastoma data.
We used H&E slides of patients with glioblastoma from the datasets TCGA GBM <cit.> and TCGA LGG <cit.>.
We filtered cases according to the latest WHO classification for gliomas <cit.>. See App. <ref> for more details.








  
Colon cancer data.
We used H&E slides of patients with colon adenocarcinoma from TCGA. The TCGA COAD dataset contains a total of 431 cases from 24 centers.




 §.§ Evaluation


All models were trained and evaluated on TCGA COAD and TCGA GBM using 5-fold nested cross validation <cit.> to allow for hyperparameters tuning and assess the generalisation independently on those datasets.

Three repeats were used in the inner loop, corresponding to three different random initializations of the MIL or CARMIL models.

As a result, metric evaluation on each of the 5 test splits was performed using an ensemble of 15 models (5 inner validation splits and 3 repeats).
Ensembling was performed by averaging the risk output of the models of an ensemble. In all the tables, we show the mean (std) C-index for OS obtained using 5-fold nested cross validation. The best results are in bold.




 §.§ Preprocessing

For tissue segmentation, a 2D U-net <cit.> trained on a pancancer dataset of manually annotated WSIs is used.

Tile features of dimension d=768 are computed using a state-of-the-art self-supervised model, Phikon, trained on pancancer H&E slides from TCGA <cit.>.

The parameters of Phikon are kept frozen in all our experiments.

The preprocessing is the same for all the models we compare to.



 §.§ Deep learning training

The Cox loss was employed in the supervised training of all models, utilizing overall survival labels. This corresponds to ℒ_MIL in eq. (<ref>).

The CAR loss, denoted as ℒ_CAR in eq. (<ref>), is applied across all CAR models and we use β=0.5 in eq. (<ref>) for the total loss accross all CAR models. 
The choice of β=0.5 followed the empirical observation that ℒ_MIL and ℒ_CAR have similar range of values during the first epoch of training.

Adam optimizer <cit.> with momentum β_1=0.9 and β_2=0.999 is used for training with a learning rate on the grid {0.001, 0.003, 0.01} for all models.
The maximum learning rate value 0.01 was chosen heuristically to be the smallest value on the grid for which most MIL models diverged during training.
The number of training epochs is optimized on the grid {20, 30}.


One NVIDIA Tesla T4 GPU with 16GB of VRAM and 8 Intel(R) Xeon(R) 2.00GHz CPUs are used for training and inference of each model.





§ EXPERIMENTS







 §.§ Survival prediction performance

In tables  <ref> and <ref>, we report the performance of various MIL models for the challenging task of survival prediction on TCGA COAD and TCGA GBM.

In table <ref>, we compare classical MIL models, selected for being agnostic to the spatial context, to their performance when enhanced with our CAR scheme.

In total, CAR improved C-index values by up to 4.8 percentage points (pp) in 9/12 settings and on average the C-index increased by 1.5 pp on TCGA COAD and 0.5 pp on TCGA GBM.

In addition, we compare classical MIL models enhanced with CAR to state-of-the-art model architectures that were designed specifically to leverage graphs of tiles. 

CARMIL models perform similarly or better than these more sophisticated models in terms of C-index; see table <ref>. 











This suggests our CAR method improves performance by successfully integrating spatial context through regularization.




 §.§ Context-Awareness performance






In this section, we compile evidence supporting the successful injection of spatial information in the proposed CAR models and its associated performance benefits. First, we assess whether CAR models genuinely exploit the input graph for their predictions. To this end, we perturb the graph at inference time by randomly shuffling all the off-diagonal terms of the adjacency matrix A. If the CAR models were to fail to exploit the graph structure, this disruption would not impact their performance. However, the results reported in table <ref> indicate that graph shuffling leads to a degradation in the model's performance, showcasing that the graph structure is indeed used in the model's decision-making process.









Secondly, we assess the Context-Awareness of trained CARMIL models compared to the feature extractor, using the DeltaCon metric that we defined in sec. <ref>. 

For each WSI, the adjacency matrix computed in the embedding space learnt by the spatial encoder of fig. <ref> is compared to the original adjacency matrix that accounts for the spatial arrangement of the tiles within the slide. 

The DeltaCon provides us with a score between 0 and 1.

A higher DeltaCon score indicates a higher degree of spatial consistency in the learnt embedding space as it implies that the arrangement in the embedding space closely aligns with the original spatial organization within the slide. We average this slide-level score across the whole TCGA COAD and GBM datasets, see table <ref>. In app. <ref>, we showcase an example of a WSI illustrating how the embeddings provided by CARMIL encoders are more spatially consistent than the original features. Moreover, in fig. <ref>, we can clearly see how the spatial encoders implement more Context-Awareness than the feature extractor (dotted lines). Indeed, DeltaCon is almost always greater for CARMIL models than for the feature extractor. Therefore, these findings confirm the successful injection of spatial knowledge resulting from the CAR. Additionally, we explored the relationship between spatial information incorporated by the spatial encoder and C-index performance. In TCGA COAD, the performance seems to correlate with the level of Context-Awareness, potentially providing insights into colon cancer. However, in TCGA GBM,  it is uncertain whether adding spatial elements improves performance, and understanding how Context-Awareness influences overall performance remains an open question.









































§ CONCLUSION


The MIL framework fails to leverage spatial interaction and organization of tiles in WSIs, potentially limiting prognostic model performance. In this work, we proposed to tackle this issue by injecting spatial knowledge into the traditional MIL framework exclusively through the prism of regularization.

We introduced CARMIL, a method to embed spatial relations between tiles directly into tile features.

This addition mimics the pathologist's consideration of spatial arrangement in slide-level prognosis.

We evaluated our method on survival prediction for colon cancer and glioblastoma, showing improved performance and revealing performance declines when spatial context is disregarded. 

We also introduced a metric for quantifying Context-Awareness, hoping to help researchers assess spatial consistency more systematically. Lastly, we discussed in App. <ref> the influence of certain parameters on our method, but we did not investigate the impact of the feature extractor on overall performance. The relationship between model robustness and Context-Awareness remains an open question, suggesting avenues for future research.






We thank  Jean Philippe Vert and Eric Durand for their thorough proofreading of our paper, as well as Jean-Baptiste Schiratti and Alexandre Filiot for providing insightful feedback on various aspects of this work. We are thankful to Céline Thiriez too for her encouragement and support.






















 












§ GLIOBLASTOMA WHO 2021 CLASSIFICATION

The WHO 2021 classification defines glioblastoma as IDH-wild type and H3-wild type brain tumor with at least one of the following features: necrosis and/or microvascular proliferation, TERT promoter mutation, EGFR amplification, or concomitant gain of chromosome 7 and loss of chromosome 10.
We will refer as TCGA GBM to those glioblastoma cases from the datasets TCGA GBM and TCGA LGG after reclassification.
TCGA GBM contains 352 cases from 18 centers.

This change of classification of glioblastoma has been shown to have a negative impact on the prognostic value of previously published biomarkers <cit.>. Therefore, it is clinically important to evaluate previous and new prognostic models on glioblastoma using the new WHO classification.



§ REPRODUCIBILITY




 §.§ Construction of the spatial adjacency matrix A


Each WSI G=(X,A) consists of a set of n tiles. For each tile p, we keep track of its spatial coordinates in the 2D plane formed by the tissue region, 𝐜_p=(x_p,y_p), in addition to the d-dimensional features vector 𝐮_p produced by the feature extractor for that tile.
We thus compute the gaussian kernel, ∀ p≠ q,

    K_pq = exp(-||𝐜_p-𝐜_q||_2/2),

and we set the diagonal to 0.
Provided k the number of neighbors, we select the k nearest neighbors for each node based on the similarity matrix K. The adjacency matrix A is defined such that A_pq equals K_pq if tile q is one of the nearest neighbors of tile p, and 0 otherwise. 



 §.§ Construction of the adjacency matrix Ã in the embedding space


Adjacency matrices in the embedding space are used for the evaluation of Context-Awareness using DeltaCon.

Consider training of our model complete and the final parameters to have converged to θ_MIL, θ_E, θ_D. Given a WSI, represented as G=(X, A), our spatial encoder returns a lower dimensional vector Z=f_E(G; θ_E)∈ℝ^n× d_E. Observe that Z_p is the d_E-dimensional embedding for tile p.
Based on this vector, we construct an adjacency matrix Ã, following the same principle as before, but this time based on the affinity between embeddings rather than using the spatial coordinates of the tiles, namely, ∀ p≠ q,

    K̃_pq = exp(-||Z_p-Z_q||_2/2),

and we set the diagonal to 0. We similarly pick the k nearest neighbors for each tile and we construct the adjacency matrix Ã, that we refer to as the adjacency matrix in the embedding space. Therefore, Ã_pq is the affinity between the embeddings of the tiles p and q.



 §.§ Computation of DeltaCon



Observe that both adjacency matrices, A and Ã, are non-symmetric. Indeed, they are derived by taking the k-nearest neighbors of each node in the kernel matrices K and K̃, which is an inherently non-symmetric operation. In simple words, tile i can be connected to tile j, but
i is not necessarly a neighbor of j, meaning the induced graphs are directed. In <cit.>, the authors introduce a proxy function defined for undirected graphs with symmetric adjacency matrix A and degree matrix D as

    S(A)  (𝕀 + ϵ^2 D - ϵ A)^-1.


We extend this definition to directed graphs by considering a non-symmetric adjacency matrix A and a degree matrix that accounts for the number of edges entering or leaving each node. Namely, considering D D_in+ D_out, where D_in, resp. D_out, is the diagonal matrix counting the incoming, resp. outcoming, edges, then we can refer back to eq. (<ref>) to generalize the definition of DeltaCon similarity between directed graphs. Given A and Ã, as described in sec. <ref> and <ref>, and their associated degree matrices D and D̃, we can now quantitatively assess the amount of spatial information, or Context-Awareness, of any CARMIL model by computing,


    DeltaCon(A, Ã) 1/1+||S(A) - S(Ã) ||_F.


In table <ref>, we assess the Context-Awareness of the best-performing CARMIL model from the 15 models evaluated through nested cross-validation, see sec. <ref>. For each WSI in each dataset, we first compute the spatial adjacency matrix A with k=8 nearest neighbors, as in sec. <ref>. Then, for all rows in table <ref> except the first, the matrix Ã is computed based on the embedding vector Z from the model's encoder after training, also with k=8, see sec. <ref>. For the first row, the adjacency matrix Ã is based on the feature vector X generated by the feature extractor. The similarity score given by eq. (<ref>) is averaged across all slides within each dataset to produce the table's values, along with the C-index mean performance of the corresponding model.
In fig. <ref>, we report these same results in a scatter plot to better illustrate two important findings. First, by comparing the scores from the feature extractor to the other models on both datasets, we confirm the successful injection of Context-Awareness into our CARMIL models. As a side note, it is worth mentionning here that, even though DeltaCon lies between 0 (indicating very dissimilar graphs) to 1 (for identical graphs), the interpolation between these extremes does not seem to evenly spread out within the interval [0,1]. This would explain the small variations in our table <ref>. In fact, as evidenced in the tables of results from the original paper <cit.>, most values are closer to 0 than 1. Nevertheless, for our analysis, only the relative ordering of these quantities is key in proving the enhanced spatial awareness provided by our method.
Second, we try to correlate gains in performance with the amount of added Context-Awareness in the CARMIL model. Conclusions  such as the more spatial knowledge is incorporated, the better the model perform, are rather difficult to draw.










 §.§ Qualitative assessment of Context-Awareness in CARMIL models


Consider a WSI, from which n tiles are sampled. Passing this slide to our feature extractor, we get a vector X∈ℝ^n× d. For each tile, we compute the mean of the eucliden distance between the d-dimensional features of this very tile with the features of all its k=8 spatial nearest neighbors within the slide. This results in a vector Y^features∈ℝ^n  that accounts for how well aligned the features learnt by the feature extractor conform to the spatial arrangement of the slide. If the features exactly reflect the spatial context of each tile, the coefficients in the vector Y^features should be fairly constant and of low magnitude. Their variations are shown in fig. <ref>, where we superposed the underlying slide with the values of Y^features.
Next, we pass these features through our CARMIL encoder at inference time, following the same procedure but using the embedding vector Z∈ℝ^n × d_E – with the same notation as before. This results in a vector Y^CARMIL∈ℝ^n. We report the coefficients in fig. <ref>.
To obtain similarly scaled heatmaps, we jointly normalize the coefficients of both matrices by their maximum value, ensuring all their coefficients range between 0 and 1. Note that both tile features X and tile embeddings Z are first normalized to have magnitude 1, ensuring a fair comparison that mitigates high-dimensional concentration effects on vector norms. 

From fig. <ref>, we can clearly see that the embeddings generated by the CARMIL encoder evolve smoothly in the 2d plane, resulting in a uniformly low heatmap value across the slide. In comparison, the tile features provided by the feature extractor Phikon exhibit less alignment with the spatial organization of the tiles, providing another qualitative indication of the enhanced spatial consideration in CARMIL models.









§ DISCUSSION ON CERTAIN PARAMETERS




  
Number of tiles n. The number of tiles n that we randomly select from the tissue region of each WSI has a significant impact, and we provide an intuitive explanation for this. The amount of spatial context required to make accurate predictions is inherently tied to the number of tiles, as it determines the scale at which patterns can be grouped. Moreover, we observe that different WSIs can vary greatly in size, resulting in a wide range of number of tiles across datasets. Whilst we fixed n regardless of the WSI at stake, it would be a natural improvement to choose n for each slide in proportion to the original size of the tissue region.



  
Number of nearest neighbors k. Similar to the number of tiles, it is probably advisable to choose k, the number of nearest neighbors, in proportion to the number of tiles n, and subsequently, the size of the WSI. In our experiments, we optimized k across a fixed grid of values, independent of n.



  
Number of GCN layers ℓ_E, ℓ_D.
The number of layers ℓ_E and ℓ_D in the spatial encoder and decoder, as well as the dimensions for the projections they induce were part of our grid search. We observed that (ℓ_E, ℓ_D)=(1,1) worked best for TCGA COAD and (ℓ_E,ℓ_D)=(2,2) for TCGA GBM. The dimensions of each layer were also finetuned from a grid of fixed values, resulting in d_E=d as an optimal choice for the spatial encoder and all intermediate layers. Besides, we observed that significantly reducing the embedding dimension d_E almost systematically led to a substantial decline in performance. 



 Paper Title: Advancing Medical Image Segmentation: Morphology-Driven Learning with Diffusion Transformer.



[
    [
    August 5, 2024
==================



§ ABSTRACT 
Understanding the morphological structure of medical images and precisely segmenting the region of interest or abnormality is an important task that can assist in diagnosis. However, the unique properties of medical imaging make clear segmentation difficult, and the high cost and time-consuming task of labeling leads to a coarse-grained representation of ground truth. Facing with these problems, we propose a novel Diffusion Transformer Segmentation (DTS) model for robust segmentation in the presence of noise.
We propose an alternative to the dominant Denoising U-Net encoder through experiments applying a transformer architecture, which captures global dependency through self-attention. Additionally, we propose k-neighbor label smoothing, reverse boundary attention, and self-supervised learning with morphology-driven learning to improve the ability to identify complex structures. Our model, which analyzes the morphological representation of images, shows better results than the previous models in various medical imaging modalities, including CT, MRI, and lesion images. Our code and dataset are publicly available at githublink






§ INTRODUCTION

Medical image segmentation is crucial in improving our understanding of complex anatomy, providing critical insights for accurate medical diagnosis and precise treatment planning. This is especially important in computed tomography(CT) scans, where the intrinsic complexity of medical images presents unique challenges that require sophisticated solutions for organ segmentation. Unlike general images, CT images are quantitative imaging, and pixel intensities are normalized to Hounsfield units (HU) values<cit.>. (e.g., air as -1000 HU, bone as +400 to +1000 HU). Therefore, clinicians must understand the quantitative meanings and select the appropriate range to enhance the visual contrast of specific tissues or organs. In particular, research is conducted to find appropriate range values for each tissue or organ in CT scans<cit.> and studies show that segmenting CT images with an inappropriate HU range normalized leads to poor performance<cit.>. ./tablesfig_drawbacks.tex This is because inappropriate normalization can occlude organs, as illustrated in Fig.<ref> (a). Additionally, clinicians can have different opinions in labeling<cit.>. Due to this, ground truths are not determinstic, and it may be difficult to obtain detailed representations of organ or lesion labels. Inaccurate manual labeling can further increase the complexity of organ segmentation, as shown in Fig.<ref> (b). We address intrinsic challenges in medical images with an architecture that combines the advantages of the adaptive and resilient Swin Transformer encoder<cit.> with the efficient decoder in UNet<cit.>. We break away from the conventional Denoising U-Net<cit.> structure because we need a model that captures a global contextual representation and can handle the various medical imaging data. In addition, we introduce three approaches to improve the segmentation process further. First, distance-aware label smoothing<cit.> is a guidance mechanism that recognizes anatomical locations in the medical image and smoothes labels by calculating location-based distances. Second, reverse boundary attention captures areas of subtle and ambiguous boundaries. This component contributes to more precise and accurate segmentation by explicitly directing the model attention to edges<cit.>, especially in the regions that have not been manually labeled. Third, self-supervised learning<cit.> allows complex features of organs to capture meaningful representations from input images in a scenario of insufficient data. We reduce reliance on labeled data and improve model adaptability to diverse and complex features of medical images. Our proposed method demonstrates generalizability beyond medical images when we evaluate it with a different domain task that can utilize morphological information. Therefore, our contribution is summarized as follows.

    

    
  * We presents a new diffusion transformer segmentation(DTS) model which performs better than previous framework(i.e. CNN based Denoising Diffusion Probabilistic Model).
    

    
  * We introduce a novel approach to address the medical image segmentation by integrating morphology-driven learning into the image processing, such as k-neighbor label smoothing, reverse boundary attention, and self-supervised learning.
    

    
  * Our model demonstrates the generality in segmentation tasks in medical modalities such as CT,  MRI, and lesion images and further suggests that this approach may be adaptable to other domains.
    





§ RELATED WORK
 

The diffusion segmentation model which applies the generative diffusion process, allows users to manipulate the ambiguity of each time step through a hierarchical structure, solving the image quality and diversity problems of existing methods, allowing the learning process to proceed stably. There is research that has notable potential applied to medical imaging<cit.>. SegDiff<cit.>, which showed consistent performance under various imaging conditions, is the first approach to solving the image segmentation problem by applying diffusion. The feature of this model is a mechanism that integrates the information of the input image and the current estimate of the segmentation map through each encoder and uses the decoder to improve the segmentation map iteratively. MedSegDiff<cit.> also applied the diffusion segmentation model to medical image segmentation. The input of the conditional image and noise segmentation map are integrated using a mechanism such as SegDiff, but high-frequency noise is constrained through the Fast Fourier transform module during the connection process. In addition, Diff-UNet<cit.> implemented the standard U-shaped architecture, which learns from the input volume in medical image segmentation effectively to extract semantic information. Here, we focus on the architecture and compare it with the existing diffusion segmentation model to demonstrate through experiments that the inductive bias, which is a major feature of CNN, can be replaced by ViT in diffusion segmentation.

Label Smoothing for Image Segmentation. Ground truth labeling for image segmentation is a time-consuming and intensive task involving experts. These processes are inherently subjective and susceptible to factors such as image quality, observer diversity, and difficulty depicting specific structures. Moreover, earlier label smoothing methods<cit.>, the inter-class relationships are usually overlooked since the labels are smoothed into one-hot encoding vectors. To address these challenges, we experimentally highlight the implementation of strategic label smoothing based on the spatial location of organs.

Reverse boundary attention refers to the integration of reverse attention<cit.>, which learns opposite concepts that are not associated with the target class in a way that substitutes existing attention mechanisms for objects, and boundary attention<cit.>, which emphasizes pixels or features of parts related to the boundary. This mechanism plays a crucial role in enhancing the performance of object segmentation in medical images. This is particularly important because medical imaging, such as CT and MRI scans, often exhibit ambiguous organ boundaries and significant amounts of noise, posing challenges for accurate segmentation. Therefore, we explore the benefits of combining unique advantages, such as a reverse boundary attention mechanism, into our framework.




§ DTS: DIFFUSION TRANSFORMER SEGMENTATION

 The diffusion model is a generative model that consists of two stages: a diffusion process and a denoising process. In the diffusion process, Gaussian noise is added incrementally to the segmentation label x_0 over a series of steps t. 
 
./tableseq_diffusion.tex
./tablesfig_architecture.tex

./tablestable_feature_extract.tex The denoising phase shown in Fig.<ref> follows the encoder-decoder network structure of standard denoising autoencoder<cit.>. As shown in Table.<ref>, we empirically suggest the possibility of replacing the latent diffusion encoder with a Swin transformer<cit.>, which has advantages such as scalability and computational efficiency when processing various images due to its hierarchical structure. Also, similar to the conditional mechanism<cit.>, our model incorporates another type of conditional encoder, τ_θ where the original image is used as input. These are demonstrated in the diffusion encoder and conditional encoder in Fig.<ref>. Our method combines information from the current estimate x_t, the image I, and the time step index t to adjust the step estimate function ϵ_θ at the input. It also takes the conditional image τ_θ(I)) and reconstructs it through a UNet decoder to produce the global feature map. Subsequently, the RBA modules facilitate the derivation of the final segmentation map, which exhibits precise edge representation, as detailed in Fig.<ref>. In conclusion, DTS(·) represents our novel diffusion transformer segmentation model, which performs segmentation by integrating the described components.



 
    ϵ_θ(x_t,I,t) = DTS((x_t,I),t, τ_θ(I))





§ MORPHOLOGY DRIVEN LEARNING


k-Neighbor Label smoothing by organ distance.
We explore medical data from body parts such as the abdomen and brain, which have organs or diseases located structurally within a compact space. As the relative positions of organs do not differ from person to person, ./tablestable_ls_scalefactor.tex we propose a k-neighbor label smoothing method that leverages the relative positions of organs for distance-aware smoothing of the labels of k neighbors for a given class or organ. In a multi-class (k>2) situation, such as in this case, there is an advantage if there is a positional relationship between them. The positional relationship refers to the relative positional relationship of organs anatomically. As shown in Fig. <ref> (b), the liver(⋆) is close to the gall bladder(▴) but relatively far from the left kidney(▪). We provided semantic information to the model based on which body structure would match this prior knowledge.  
./tableseq_k-nls.tex
./tablesfig_k-nls.tex

./tablesfig_RBA.tex
RBA: Reverse-boundary Attention.
Complex anatomy and the inherent ambiguity in defining boundaries of adjacent organs are factors that hinder accurate segmentation of organ boundaries in medical images. Considering that these factors are likely to result in false positives or missing details in the initial segmentation, our approach includes selectively dropping or reducing the prediction weights of overlooked regions. The Reverse Boundary Attention method aims to improve the prediction of segmentation models by gradually capturing and specifying areas that may have been initially ambiguous. Thus, our architecture removes previously estimated predictive areas from high-level output features where existing estimates are upsampled in deeper layers, sequentially explores these details, including areas and boundaries, and finally, improves the segmentation model predictions progressively.
./tableseq_RBA.tex

./tablesfig_ssl.tex
Self-supervised learning (SSL)
 can encode anatomical information of the human body in the image effectively. We propose three proxy tasks for learning comprehensive semantic representations within masked images without using labels. Our framework combines (1) Contrastive learning(e.g. SimCLR<cit.>), which encodes masked images to improve the ability to distinguish between different samples with hidden feature representations; (2) Masked Location Prediction, which predicts the location of the samples; and (3) Partial Reconstruct Prediction(e.g. SimMIM<cit.>), which learns the feature representation by reconstructing the masked patch area of each sub-volume.  These widely recognized self-supervised learning strategies are both straightforward and effective.

 When the input(demonstrated 2D image in Fig.<ref>) is divided into patches and then passed as input to the encoder twice, two sets of latent embeddings are obtained, and a contrastive learning is performed through constrastive loss<cit.> (Eq.<ref>). Then, masked location prediction is conducted to predict the number of randomly masked parts by dividing the ĥ_̂î image into patches from 0 to 8 (Eq.<ref>).
In addition, Partial Reconstruct Prediction is performed by masking the image of ĥ_̂ĵ, reconstructing it through a decoder, and learning the difference from the original through L_2 loss (Eq.<ref>).

./tableseq_ssl.tex




§ EXPERIMENTS

Datasets. The pre-training dataset consists of medical images sourced from partial accessible CT, MRI datasets encompassing 3,358, 6,970 subjects respectively. Notably, the pre-training step does not involve the utilization of annotations or labels on this dataset. The primary objective of this pre-training process is to enable the model to learn meaningful representations from the available image data, thus eliminating the need for manual annotation. 
The BTCV<cit.> dataset comprises 3D abdominal multi-organ CT images from 30 cases, each associated with a specific form and featuring 13 multi class segmentation objectives.
The BraTS2021<cit.> dataset includes 1,251 subjects of brain MRI images. Each image is annotated with three segmentation targets and encompasses four modalities T1, T1Gd, T2, and T2-FLAIR. 
The ISIC2018<cit.> dataset contains 2,594 dermoscopic images of skin lesions, each annotated by experts for segmentation purposes. 
The Cityscapes<cit.> dataset contains several urban street scenes for segmentation purposes and is used to test the generalization performance of our approach. It consists of 3475 semantically annotated train, val sets and 1525 test set.
Details about datasets are shown in the appendix.

Implementation Details.
Our architecture implemented in PyTorch and MONAI[https://monai.io/https://monai.io/]. For pre-training tasks, the reconstruction strategy is applied with a mask ratio of 0.4. Moving on to the fine-tuning phase, the AdamW optimizer<cit.> is used with a weight decay 1e^-3. The warm-up is set to 0.1 of the total epochs, and the learning rate undergoes linear updates following the Cosine Annealing schedule<cit.>. The loss function incorporates DICE loss<cit.>, BCE loss, and MSE loss. Random flips, rotations, intensity scaling, and shifts were applied to augment the data. We set the number of diffusion steps as 1000, and the sliding window overlap rate is 0.8 until the final prediction. Preprocessing details for each dataset are provided in the appendix.

Evaluation Metrics are important to quantify the performance of the segmentation model. Two commonly used metrics are the dice similarity coefficient<cit.>(Dice) and the Hausdorff distance<cit.>(HD). The evaluation metric are as define. Y and Ŷ represent the actual and predicted values in input units, and g' and p' represent the actual and predicted values of points on the surface.
./tableseq_metric.tex

./tablestable_ls_ablation.tex
Exploring the performance of Label Smoothing
We concentrate on the performance of k-neighbor label smoothing and explore its applicability to general datasets with structural properties. We explore its applicability to a cityscapes<cit.> dataset with structural properties by utilizing only our baseline model and label smoothing. Compared with basic label smoothing(uniform), Non-Uniform Label Smoothing(NULS), and especially Spatially Varying Label Smoothing(SVLS), which applies label smoothing to neighboring pixels using weight matrix in the form of Gaussian kernel, we can see that our performance is superior. Previous methods compensate for the label’s uncertainty in image segmentation, but our methods further estimate the positional relationship between classes to improve prediction performance with a label smoothing method, emphasizing that this can be easily applied to other tasks.

./tablestable_ssl_ablation.tex
Efficiency of Self Supervised Objectives. We conduct comprehensive ablation experiments on the BTCV dataset to evaluate the efficiency of self-supervised learning. In these experiments, we employed specific settings for calculating the loss, and the obtained results are presented in the Table.<ref>. Notably, the L_Rec is learned based on the pixel representation of input images,  L_Loc (Masked Location Prediction) is learned by recognizing the location of the masked region, and L_CL (Contrastive Learning) is focused on contrastive learning at two augmented sample level. The L_Rec lies in its important role in understanding meaningful representation learning from medical images, as shown in experimental results. By employing these three loss functions, our self-supervised learning approach aims to capture intricate details at both pixel and region levels, enhancing the model's ability to extract meaningful features from the the inputs.

./tablestable_architecture_ablation.tex
Selecting the optimal architecture
Remind the our approaches, in the case of self-supervised learning (SSL), feature representations pre-trained from the three proxy tasks are transferred to the conditional encoder to assist in understanding the original image. We experiment with the effect of freezing or leaving all weights trainable during benchmark fine-tuning. Additionally, our framework explores the ablation study on reverse boundary attention, which is integrated with the general diffusion segmentation process. We comprehensively verify the effectiveness of morphology-driven learning within the architecture to prove its hypothesis. As shown in the Table.<ref>, the single module experiments(presented in the second section) show higher performance than the scratch model, but learning by leaving the conditional encoder trainable shows a large margin in the BTCV dataset. This indicates that feature representation was achieved by aligning the learned features well with the downstream task. Our model, which comprehensively combines morphology-driven learning techniques, shows remarkable improvement in results, and our final architecture is shown in Fig. <ref>.

./tablestable_quan_result_btcv.tex
./tablesfig_qual_result_btcv.tex



§ COMPARATIVE RESULTS

As shown in Table.<ref>, we compare our model with the BTCV benchmark dataset. Compared with other models, the proposed DTS achieves the best performance and presents a higher dice result of 0.906. It can be seen that previous diffusion segmentation models show comparable performance to conventional segmentation models in relatively large organs(e.g. liver, stomach), but poor performance in small organs(e.g. esophagus, aorta). DTS surpasses the closest competing methods by an average of 2% across all classes, with an even more significant improvement of 7% specifically for gall bladder. We believe that our approach and the application of the high performance transformer architecture will lead to improved accuracy. Comprehensive qualitative results of our model, which demonstrate good segmentation performance for small organs, can be found in Fig.<ref>, highlighting our model's ability to capture details and achieve accurate boundary representations.

./tablestable_quan_result_brats_isic.tex
The results presented in Table.<ref> demonstrate that the two datasets showed optimal outcome with an average accuracy in terms of both Dice and HD score. In particular, within the ISIC dataset, solely K-neighbor label smoothing was omitted from the application. This decision was made due to the dataset has only a single label without structural position relationships between adjacent labels. Consequently, employing the K-neighbor label smoothing method in this specific scenario is unnecessary. Overall, SwinUNETR <cit.> has a competitive performance in the benchmark results. Although it uses a similar architecture as DTS, we believe that it does not lead to the robustness and of diffusion models, which excel against noise and artifacts in the input data, particularly in medical images.




§ FUTURE WORK AND CONCLUSION

Our study focuses on the advantages of morphology-driven learning for segmentation tasks, where our approach demonstrates substantial improvements. Building on these promising results, we aim to broaden the scope of our framework by applying it to other critical imaging tasks, such as classification and detection, to evaluate its effectiveness across various domains and imaging scenarios.
In conclusion, we present a novel approach to medical image segmentation. DTS suggests the potential to replace existing CNN-based down-sampling by using a Swin Transformer encoder. We believe that this model architecture enables accurate segmentation with small, detailed representations and improves performance by complementing the chronic problems of medical images with Morphological-based learning, such as k-neighbor label smoothing, reverse boundary attention and self-supervised learning. We hope that this inspires future tasks in situations with morphologically complex problems.







§ ACKNOWLEDGEMENTS

This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2024-2020-0-01789), and the Artificial Intelligence Convergence Innovation Human Resources Development (IITP-2024-RS-2023-00254592) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation). Additionally, this work has been carried out with (extensive) use of the NEURON computing resource that is supported by the Korea Institute of Science and Technology Information (KISTI).




 Paper Title: VITAL: Visual Teleoperation to Enhance Robot Learning through Human-in-the-Loop Corrections.


Bimodal Scaling Law and Size Effect In Superelastic Nanopillars
    Xian Chen
    August 5, 2024
===============================================================




§ ABSTRACT 
   Imitation Learning (IL) has emerged as a powerful approach in robotics, allowing robots to acquire new skills by mimicking human actions. Despite its potential, the data collection process for IL remains a significant challenge due to the logistical difficulties and high costs associated with obtaining high-quality demonstrations. To address these issues, we propose a low-cost visual teleoperation system for bimanual manipulation tasks, called VITAL. Our approach leverages affordable hardware and visual processing techniques to collect demonstrations, which are then augmented to create extensive training datasets for imitation learning. We enhance the generalizability and robustness of the learned policies by utilizing both real and simulated environments and human-in-the-loop corrections. We evaluated our method through several rounds of experiments in simulated and real-robot settings, focusing on tasks of varying complexity, including bottle collecting, stacking objects, and hammering. Our experimental results validate the effectiveness of our approach in learning robust robot policies from simulated data, significantly improved by human-in-the-loop corrections and real-world data integration. Additionally, we demonstrate the framework's capability to generalize to new tasks, such as setting a drink tray, showcasing its adaptability and potential for handling a wide range of real-world bimanual manipulation tasks. A video of the experiments can be found at:  
    




 





§ INTRODUCTION


Imitation Learning <cit.> has emerged as a fundamental approach in robotics, enabling robots to acquire new skills by observing and mimicking human actions <cit.>. This method holds significant promise for developing complex robotic behaviors without the need for explicitly programming the robot. However, the primary challenge with imitation learning lies in the data collection process <cit.>. On the one hand, acquiring high-quality demonstrations is costly and logistically challenging. On the other hand, the need for extensive and diverse demonstrations to train models magnifies these difficulties, as each new task or environment often requires a fresh set of demonstrations <cit.>. To address these challenges, teleoperation has been proposed as a viable solution. In recent years, there has been a growing interest in the development of bimanual teleoperation interfaces that facilitate the learning and execution of various household and industrial tasks <cit.>. 


One example of such a system is the ALOHA platform <cit.>, which has garnered considerable attention from the robotics research community. Such teleoperation platforms enable the execution of various tasks such as pushing chairs, using cabinets, wiping spills, and even cooking shrimp. Despite the advantages offered by such interfaces, there remain significant challenges that hinder their widespread adoption. One of the primary issues is that such teleoperation interfaces are hardware-specific and costly. The cost of such interfaces is approximately $30K, which makes them non-scalable and expensive for many research laboratories and practical applications. To overcome these challenges, we propose a low-cost visual teleoperation interface to collect high-quality demonstrations for long-horizon bimanual manipulation tasks.


In robot manipulation tasks, IL is often trained via in-domain demonstrations, which correspond to the data obtained directly from the deployment environment <cit.>. Although it is widely accepted that gathering direct examples from the actual robot is the most efficient way to learn long-horizon manipulation tasks, we show that collecting human demonstrations in real and digital twin environments can produce better outcomes for real-world tasks. Our approach allows the creation of a large dataset of augmented demonstrations based on a few initial demonstrations. By leveraging the flexibility and functionalities of our digital twin environment, we can enhance the number of demonstrations and the generalizability of the learned policy. These augmented demonstrations are then used to train a policy, which is fine-tuned using real-world demonstrations. Experimental results show that our approach significantly improves the performance of the robot in real-world scenarios. In summary, the main contributions of this work include:


    
  * We introduce a low-cost visual teleoperation approach that enables high-quality data collection for bimanual manipulation tasks.
    
  * We leverage a digital twin of the robot to create a vast dataset of augmented demonstrations from a limited number of initial demonstrations. This enhances the scalability and generalizability of the learned policies without requiring extensive real-world demonstration.  
    
  * We incorporate human-in-the-loop corrections to refine and improve the robot's performance in complex tasks. This method allows for interactive learning and adaptation, ensuring higher success rates in real-world applications.





























§ RELATED WORK

Robot Learning with Teleoperation: IL is a powerful technique for robot learning, enabling robots to acquire new skills by mimicking human demonstrations <cit.>. However, collecting in-domain demonstrations is often difficult and not scalable due to the need for physical interaction with the robot hardware. This process is also expensive as it requires constant access to the robot. Some works use scripted “expert” in simulation to generate demonstrations, but scaling these approaches to more complex tasks is challenging <cit.>. Other IL methods leveraged visual deep neural networks to train reactive models from images captured during demonstrations <cit.>. While these models provide greater flexibility, they necessitate collecting a large number of demonstrations to learn simple pick-and-place tasks, making them user and computationally expensive <cit.>. Other approaches utilize human activity videos to guide robot learning but often fall short in training 6-DoF manipulation policies <cit.>. These methods typically require in-domain teleoperation data to bridge the gap between video demonstrations and robotic execution <cit.>.
A common problem with these IL methods is the lack of corrective feedback, leading to the accumulation of errors over the length of the demonstration, known as the compounding errors problem <cit.>. Similar to our approach, some methods have focused on dividing demonstrations into several waypoints to address this issue <cit.>. Unlike these approaches, we aim to automatically extract the waypoints of a demonstration without adding any burden on the human user. Other IL methods used bimanual teleoperation interfaces, such as ALOHA <cit.>, for mapping joints' value from master to slave robots. However, these approaches are quite expensive and not scalable as they require four robots for bimanual manipulation: two masters and two slaves. In contrast, we propose a low-cost teleoperation interface that allows controlling robot end-effectors to perform complex bimanual tasks and collect demonstrations in both real and simulation environments. Teleoperation within a digital twin of the robot enables us to scale up the training data easily without constant access to the real robot. Additionally, we introduce a human-in-the-loop correction mechanism to repair faulty demonstrations, enhancing the robot's performance in complex tasks with minimal effort. 

Sim2Real Transfer: Some approaches, such as MimicGen <cit.> and CyberDemo <cit.>, attempt to collect demonstrations in simulation to reduce the cost of collecting demonstrations on robots. However, due to the sim-to-real gap, deploying the learned policy on a real-world robot remains challenging. Other methods use VR for teleoperation and data collection <cit.>, such as DIME <cit.>. Qin et al. <cit.> and Wang et al. <cit.> proposed tracking the user's figures through an external camera for teleoperation. Approaches like DexMV <cit.>, DexVIP <cit.>, and VideoDex <cit.> use everyday videos to learn motion priors. Due to the domain gap, these methods are not directly applicable to robots and often require another round of training.
In contrast, we collect demonstrations in the digital twin of the robot and develop data augmentation techniques at the trajectory level rather than the image data level. Most of the teleoperation interfaces have been designed either for simulation <cit.> or for real robot  <cit.> setups. Unlike these approaches, we proposed a low-cost bimanual teleoperation interface that can be used to collect demonstrations using real-robot and its simulation. We showed that policies trained on a mix of simulated and real demonstrations achieve better results than those trained solely on simulation or real-robot data. 





§ METHOD





 §.§ Collecting Demonstrations through Teleoperation

r0.4


    
    < g r a p h i c s >

    Overview of the proposed low-cost teleportation interface.
    
    


    
The architecture of our low-cost visual teleoperation system is illustrated in Figure <ref>. 

We use an external RGB-D camera to capture human motions and two Bluetooth-enabled selfie sticks for commanding the left and right grippers. The captured human motion data and gripper commands are translated into real-time motor commands for the robot arm and the gripper. In particular, we utilized the Mediapipe library for skeleton tracking, specifically focusing on 24 upper body keypoints. These keypoints are crucial for capturing the operator's movements accurately. To establish a stable huamn-world reference frame, we considered the left shoulder, right shoulder, left hip, and right hip keypoints. We define the origin of this reference frame between left and right hip keypoints. This reference frame serves as the basis for calculating the pose of each hand relative to the world frame. Given the differences in body morphology and arm length between humans and the robot, two scaling factors, α for the right arm and β for the left arm,  are applied to bridge the gap, thereby making teleoperation more intuitive and precise. For controlling the grippers of the robot's arms, we utilized two Bluetooth selfie sticks. These sticks were adapted to send commands for opening and closing the grippers. 
r0.15
    

    
    < g r a p h i c s >

    Low-cost teleportation sticks.
    

    

To track the sticks' orientation, we attached AruCo markers <cit.> to the sticks as shown in Fig. <ref>.



To efficiently manage the complexity of long-horizon tasks and reduce the compounding errors, we employ task decomposition. Each task, T, is decomposed into a sequence of subtasks, S_n, T = (S_1, S_2, …, S_n). The start and end poses of the trajectory for each subtask are defined based on the current pose of the manipulator and the pose of the object that should be grasped or placed. 
 
The detection of these subtasks is facilitated by the commands given by the user for grasping or placing objects. When a user issues a command, the system identifies the relevant subtask, captures the necessary pose information, and creates a trajectory segment for that subtask. By structuring tasks this way, we ensure that the demonstration data is organized and can be effectively used for training purposes.

To replicate the physical environment in the simulation, we build a digital twin of our robot in Gazebo. This includes accurately modeling the robot and objects to ensure consistency between the simulated and real-world environments. During the demonstration phase, we performed a task in Gazebo while recording all relevant ROS topics, including the initial state of the objects, joint values, end-effector poses, outputs from RGB-D cameras (head-mounted and on the left and right arms), commands from the instructor, activity videos with corresponding 3D skeleton data, object poses, and success or failure indicators, at their respective publication rates into a ROS bag file. The comprehensive nature of this recorded data ensures that all necessary information is captured for the next stage of the methodology, where data augmentation will be applied to enhance the dataset.






 §.§ Augmenting Demonstration in Simulation


To enhance the collected demonstration and improve the robustness of the learned policies, we perform data augmentation on the trajectory level (i.e. subtasks). Our augmentation method uses several techniques to generate a diverse set of trajectories from the initial demonstrations. First, we extract waypoints from the recorded bag file by reading the positions and orientations of the robot's end effectors. These waypoints are then used to fit a polynomial trajectory. We exclude the last few points from the polynomial fitting to ensure the end of the trajectory which represents the affordance pattern is preserved. We then apply uniform sampling to the trajectory by calculating the cumulative arc length of the waypoints to ensure uniform spacing between the sampled points. 
This method helps in creating a smooth and evenly distributed trajectory, which is then segmented into smaller parts for detailed analysis and augmentation.

We first augment the trajectory by adding Gaussian noise to the waypoints. This involves introducing random variations to the trajectory points to simulate real-world imperfections and variations. By adding noise, we can create multiple versions of the trajectory, enhancing the model's ability to generalize across different scenarios. The noise can be applied to all waypoints or selectively to preserve certain critical points, such as the trajectory's endpoint.
We also perform translations and flips on the trajectory. Translating the waypoints involves shifting the entire trajectory in the x, y, and z directions, which helps in creating varied instances of the task. Flipping the trajectory along the x-axis generates a mirrored version, providing additional diversity in the dataset and making the initial trajectory of one arm useful for the other arm. Furthermore, we sample new initial points within specified bounds and connect these points to the nearest waypoints in the existing trajectory. This process involves finding the nearest index in the trajectory and creating new paths from these initial points to the rest of the trajectory. By varying the starting points, we can simulate different initial conditions for the task. Each trajectory is annotated with an arm identifier, distinguishing between left and right arm movements. This comprehensive augmentation process significantly expands the dataset, making it thousands of times larger than the original set of demonstrations. However, augmented trajectories can sometimes be imperfect and not executable due to the robot's arm kinematics, potentially leading to task failures. To ensure the reliability of the augmented data, we validate each augmented trajectory in the digital twin environment. Only successfully executed trajectories are used for training a policy. This way we ensure that the learned policies are robust and capable of handling a wide range of real-world conditions.






 §.§ Hierarchical Policy Learning




To effectively learn and execute long-horizon tasks, we formulate the learning policy as a hierarchical policy learning problem through Behavioral Cloning (BC). This approach involves training a high-level policy to select a sequence of subtasks and low-level policies to execute each subtask. 

For a given task, we assume that we can detect the sequence of subtasks based on the demonstration by detecting commands for grasp or release (indicated by pressing the bottom of the selfie stick). These commands allow us to identify the start and end poses for each subtask, facilitating the decomposition of long-horizon tasks into a series of subtasks.

The high-level policy, π_H, is responsible for selecting the appropriate subtask, S_i, based on the current state of the environment and the task progression. This high-level policy can be implemented as a state machine or a learned model. In this work, we use a state machine that receives the current state of the robot, and the pose of the objects to infer the current subtask, π_H(o, r)→ S_i, where o shows the current state of objects and r is the state of the robot. Alternatively, a learned model could receive the task ID and the state of the objects and robot, and predict the subtask.



For the low-level policy, we propose a unified multi-subtask policy learning, π_MS, that can handle all subtasks. This policy receives the predicted subtask ID, S_i, along with the start, p_s, and end poses, p_e, and generates the appropriate trajectory conditioned on the input subtask, π_MS(S_i,p_s, p_e) → trajectory. The training process begins by loading the augmented trajectory data of a task. The trajectories are padded to ensure uniform length, which is critical for batch processing in neural networks. Our model is designed to predict the intermediate points of a trajectory given the start and end points. The high-level policy coordinates these subtasks, ensuring smooth transitions and overall task completion.










 §.§ Residual Learning with Human-in-the-Loop


r0.5
    


    < g r a p h i c s >

    Overview of the residual learning based on human-in-the-loop feedback.
    
    


Despite the robustness of the trained policy, there can be scenarios where the robot fails to complete the task due to a sim-to-real gap, pose estimation errors, or small errors in executing a trajectory. To address these issues, we incorporate a residual learning approach with human-in-the-loop corrections during the policy rollouts. When the robot encounters a failure during task execution, a human operator steps in to correct the errors. The user plays the recorded rosbag file of the experiment and provides the correction feedback whenever it is needed. The correction involves measuring the 3D delta position changes of the wrist of the left and right hands of the demonstrator relative to the initial poses of the hands. These delta changes are then scaled by empirically determined factors α (for the right hand) and β (for the left hand), both < 0.1 to avoid fast movements, and added to the robot's wrist movements. This minimal adjustment helps the robot to accomplish the task with improved accuracy. If the user observes a large error in the current prediction of the policy, they can switch the system to full teleoperation mode. In this mode, the robot will directly follow the human wrist movements to accomplish the task. 

The data from these human-in-the-loop corrections is recorded into a separate dataset, 
D'. This dataset includes both residual corrections and instances where full teleoperation was required. To fine-tune the policy, training data is sampled equally from both the original dataset D and the new dataset D'. This balanced sampling ensures that the model learns from the augmented experiences and corrections provided by the human operator, thereby improving its performance and adaptability. This iterative process of learning ensures that the policy continuously improves and adapts to real-world conditions.




§ EXPERIMENTAL SETUP



We performed multiple rounds of experiments on both simulation and real-robot settings to validate our method. Specifically, we study the following questions: (Q1) Is it feasible to train robot policies using exclusively simulation data without relying on on-robot data? 
(Q2) Which model architectures and data composition ratios are most effective for policy training? Specifically, what is the optimal balance between simulated and real-world data?
(Q3) To what extent does human-in-the-loop correction enhance policy performance when teleoperation data is limited?
(Q4) Can our framework successfully execute another complex bimanual task, such as setting a drink tray, which is somewhat similar to one of the learned tasks?



 §.§ Setup
 
Our experimental setups, both in simulation and on real robots, are illustrated in Fig. <ref> and Fig. <ref>, respectively. We developed a simulation environment in Gazebo, leveraging the ODE physics engine to closely mimic the behavior of our dual-arm robot. The hardware setup in both simulation and real robot includes an Asus Xtion camera, two Universal Robots (UR5e) equipped with Robotiq 2F-140 grippers, and an interface for managing the initiation and conclusion of experiments. For the Teleoperation interface, we use a RealSense D435 RGB-D camera to capture the motion of the user. All evaluations were performed with a PC running Ubuntu 18.04 with a 3.20 GHz Intel Xeon(R) i7, and a Quadro P5000 NVIDIA. 

r0.38


    
    < g r a p h i c s >

    Three long-horizon evaluation tasks are used. (left) Bottle Collecting; (center) Stacking Pringles; (right) Hammering.
    

    

Tasks: We designed three tasks of varying difficulty levels to test the performance of the teleoperation and the robustness of the learned policies: (i) Bottle Collecting: In this task, the robot should collect multiple bottles with different shapes from different locations. (ii) Stacking Pringles: In this task, the robot must stack Pringles cans on top of each other in various locations. This requires precise control and coordination to ensure the cans are aligned and stable. 
(iii) Hammering: The most challenging task involves the robot using a hammer to strike a cylinder. This task tests the robot's ability to handle a tool effectively. To evaluate the performance of the robot on each task, we performed 100 simulation and 10 real-robot experiments. In the case of real robot experiments, we used a 3D perception system to get the pose and label of the objects <cit.>.
r0.38


    
    < g r a p h i c s >

    (left) Our real dual-arm robot setup; (right) Output of our perception including the bounding box, pose, and label of the objects.
    

    





Baselines and Metrics: To determine the best model architecture for bimanual manipulation tasks, we evaluated multiple Behavioral Cloning Recurrent Neural Network (BC-RNN) baselines. The baselines include: LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), and Transformer-based network. PyTorch is used as a deep learning framework. We use Mean Squared Error (MSE) as the loss function for trajectory prediction. An Adam optimizer is employed to adjust the model weights during training. Early stopping is implemented to halt training if the validation loss does not improve for a specified number of epochs. Each network has been trained for 200 epochs. We split the data into training (70%), validation (15%), and test sets (15%) to evaluate the model's performance. Details of the networks are discussed as a part of supplementary materials. The Task Success Rate is used as the primary metric for evaluation. This metric is defined as the percentage of successfully completed tasks out of the total number of attempts. 





 §.§ Results



Q1 - Feasibility of Training Policies with Simulation Data:
We conducted extensive sets of experiments across all tasks by training the policy solely on augmented simulation trajectories. For each task, we collected five successful in-domain demonstrations and augmented them to 80K demonstrations. The BC-LSTM policy trained on this augmented data was then tested on both simulated and real robots. Results are summarized in Fig. <ref>. In simulation, the robot successfully performed most tasks. However, in real-world experiments, 
r0.38


 
    < g r a p h i c s >

    Results of Q1 evaluation.
    

    

certain discrepancies such as minor inaccuracies in object grasping and placement were observed. Specifically, in the bottle collecting task, the robot achieved an average task success rate of 92% in simulation and 80% in real-robot scenarios. In the stacking objects task, the robot successfully completed 84% of the experiments in simulation but only 60% in real-world tests. Similarly, for the hammering task, the success rate was 78% in simulation and 50% in real-world experiments. The main failure in bottle collecting was due to inaccurate trajectory prediction, which led to inefficiencies in approaching, grasping, or placing the bottles. The primary reasons for failures in the stacking and hammering tasks were inaccuracies in object grasping. Specifically, when the robot approached to grasp the object, the tip of the gripper touched the object, causing it to move slightly. This offset often led to failures as objects falling during the last step of stacking or missing the target when hammering. To address these challenges, we will check the impact of the model architecture and the effect of incorporating real demonstrations in addition to digital-twin demonstrations for training the policy to address sim-to-real gap in the following subsection (Q2). Furthermore, we believe incorporating feedback mechanisms through vision sensing could provide information on the exact part of the object being grasped. This feedback would enable the system to update the trajectory accordingly by learning suitable residual corrections, thereby improving the accuracy and reliability of the task execution. 













Q2 - Impact of Model Architecture and Data Composition: To understand the influence of different model architectures and the ratio of simulated to real data, we conducted experiments with various configurations. We first trained LSTM, GRU, and Transformer-based models solely on augmented simulation demonstrations and compared their performance across the proposed tasks in the simulation environment. After identifying the best-performing architecture, we trained policies using different ratios of simulated to real-robot demonstrations, including 70%-30%, 50%-50%, 30%-70%, and 0-100% mixes. For each task, we recorded five real-world demonstrations and five simulation demonstrations, which were then augmented to create a larger dataset. The final training data was sampled from these augmented demonstrations according to the specified ratios. This approach allowed us to determine the optimal balance between simulation and real-world data for training robust and effective robot policies.
The LSTM model provided a good balance of performance and training efficiency, whereas GRU models exhibited slightly lower performance but benefited from reduced computational overhead. Transformer-based models demonstrated considerable potential for handling more complex tasks; however, they demanded significantly more computational resources (see Fig. <ref>). 
r0.38


    
    < g r a p h i c s >

    

    Results of Q2 evaluation.
    

    

Upon comparing the results, we identified that the optimal ratio of simulated to real data was 70% simulated and 30% real. This ratio offered the best generalization and performance in real-world tasks, enhancing the task success rate across all evaluated tasks. Specifically, for bottle collecting, the task success rate increased from 80% to 90%; for stacking Pringles, the success rate rose from 60% to 80% (see Fig. <ref>); and for hammering, the success rate improved from 50% to 70%. This balanced approach effectively mitigated the sim-to-real gap, thereby enhancing the robustness and reliability of the learned policies.

Q3: Effectiveness of Human-in-the-Loop Corrections:
In this round of experiments, we investigated the effectiveness of human-in-the-loop correction in scenarios when the policy failed during task execution. We recorded 5 failure experiments for each task in the simulation. We then replayed the bag file and allowed the human user to correct the robot’s movements using residual learning or full teleoperation mode. The updated demonstrations were tested to ensure they could successfully accomplish the task. These corrected demonstrations were then augmented 
r0.38


    
    < g r a p h i c s >

     

    Results of Q3 evaluation.
    

    

and used for policy fine-tuning. By comparing the results, we observed that human-in-the-loop corrections significantly improved task success rates, particularly for more complex tasks such as stacking and hammering. 

The inclusion of human corrections from dataset D', resulted in an average improvement of approximately 5% in task success rates, demonstrating the value of interactive learning. Specifically, the success rate for the hammering task increased from 78% to 86%, for the stacking task from 84% to 89%, and for the bottle collecting task from 92% to 95%. These results indicate that the policy was effectively learning from the corrections, as the number of correction feedback by the human user decreased over time. 

 

r0.38
    

    
    < g r a p h i c s >

    An example of Setting a Drink Tray task: (left) initial and (right) desired objects pose.
    

    

Q4: Capability to Execute A New Complex Bimanual Task: Setting a Drink Tray
To evaluate the generalization capabilities of our framework, we tested its ability to execute a new bimanual task that shares similarities with the learned tasks: setting a drink tray (see Fig. <ref>). This task involves placing a cup and a bottle on specific markers on a tray. The tray has two circular markers: one red for placing the cup and one blue for placing the bottle. The robot must pick up a cup and a bottle from a starting position and place them accurately on the designated red and blue markers on the tray. The policy was initially trained on the previously learned object stacking task. The trained policy was then evaluated only in simulation, as we could obtain precise positions of the markers and objects. The task was considered a success if the robot accurately placed the cup and bottle on the respective markers and a failure if the distance between the object and the marker exceeded 3 cm. We repeated this experiment 
100 times, with the objects placed in random positions each time. The robot achieved a task success rate of 87%. The primary source of error in the failure cases was slight misalignment during grasping, which led to minor placement inaccuracies, despite accurate trajectory predictions. Our approach successfully generalized to the new task, demonstrating that the learned policies could be adapted to similar but distinct tasks with minimal effort. This showcases the robustness and adaptability of our teleoperation and learning framework in handling a variety of real-world scenarios. We believe that incorporating feedback mechanisms through vision, such as detecting the exact positions of the markers and adjusting the trajectory accordingly, could further enhance the accuracy and reliability of task execution.





§ CONCLUSION AND LIMITATIONS



In this work, we presented a low-cost visual teleoperation system for bimanual manipulation tasks. Our approach leverages an RGB-D camera and visual processing techniques to collect demonstration data, which is then augmented and used to train robot policies. We employed a combination of simulated and real-world data, augmented with human-in-the-loop corrections, to improve the accuracy and reliability of the learned policies. Our system was evaluated on three tasks with differing complexity levels, demonstrating its effectiveness in both simulated and real-robot environments. The experimental results validate the effectiveness of our approach in learning robust policies from simulated data, enhanced by human-in-the-loop corrections, and real-world data integration. The framework successfully generalized to a new, complex task —setting a drink tray—demonstrating the adaptability of the learned policies to similar tasks with minimal effort. 

The main limitation of this work is that for tasks requiring a high degree of precision at certain steps (e.g., staking, hammering, etc), waypoint trajectory learning may not be sufficient, and (closed-loop) vision feedback should be considered. To address this limitation, we plan to explore the integration of visual feedback mechanisms to further enhance the accuracy and reliability of task execution. By incorporating real-time visual feedback, the system can adjust and refine its actions based on the visual context, leading to more precise and robust manipulation capabilities. 








  





§ APPENDIX






§ IMPACT OF MODEL ARCHITECTURE AND DATA COMPOSITION (Q2)




















   


































 §.§ Training data composition

We trained several policies using different ratios of simulated to real-robot demonstrations, including 70%-30%, 50%-50%, 30%-70%, and 0-100% mixes. For each task, we recorded five real-world demonstrations and five simulation demonstrations, which were then augmented to create a larger dataset. The final training data was sampled from these augmented demonstrations according to the specified ratios. This approach allowed us to determine the optimal balance between simulation and real-world data for training robust and effective robot policies.







§ VISUALIZATION OF GROUND-TRUTH AND PREDICTED TRAJECTORIES 
 


We learned a policy using a single-shot demonstration and tested its performance on the validation set of the first subtask of the bottle-collecting task. Specifically, we plot 2D/3D visualization of predicted and actual trajectories for two randomly selected samples from the validation set. The network's performance is depicted through both 2D and 3D plots in Figure <ref>. Furthermore, we tested the performance of the policy on the digital twin of the robot by randomly placing a bottle object in front of the robot, retrieving the current pose of the end effector and the pose of the object, and forwarding them as input to the learned policy. We then executed the predicted trajectory and visualized the robot's performance. 

Figure <ref> shows the performance of the robot in three experiments, highlighting its ability to follow the predicted trajectory and accurately grasp the object based on the learned policy. The visualizations include both images of the robotic arm and plots of the predicted versus actual trajectories for the X, Y, and Z axes.



§ EXPERIMENTS



 §.§ Success and failure criteria for the proposed tasks


r0.35


 
    < g r a p h i c s >

    Sucesses and failures for stacking and hammering tasks.
    
    



For the collecting objects task, success is determined by placing all objects into the basket and failure is considered if one of the objects is not placed into the basket and remains outside. Figure <ref> shows examples of successful (left) and failed (right) attempts for the stacking and hammering tasks. In the stacking task, success is achieved when objects are stacked stably. In the hammering task, success criteria are based on hitting the target object accurately.  These criteria help in evaluating the performance and reliability of the robotic system in executing these tasks. In the case of setting up a drink tray task, success is considered when the robot places all items (such as a mug and a bottle) accurately on their designated spots on the tray, and failure is defined when items are not placed correctly on the tray, or the robot fails to place all items.






 §.§ Performance of the robot in simulation: known scenarios - known objects

We show the performance of the robot in the simulated environment where the robot interacts with known objects across known scenarios in Figure <ref>. The tasks include collecting bottles (top-row), stacking objects (second-row), and hammering (lower-row), all using the same objects from the collecting demonstration. The robot's ability to follow a predicted trajectory and complete these tasks accurately showcases the robustness of the learned policy.






 §.§ Performance of the robot: known scenario - new objects


In this section, we test the robot's ability to adapt to new objects in a familiar scenario. The experiment involves using bottles that differ in weight, size, and shape from the ones used in the initial demonstration. This assessment helps determine the robot's generalization capability when faced with varying object properties while performing the same task. Despite the differences in the object's physical characteristics, the robot successfully follows the predicted trajectory and completes the collection task. The sequence of snapshots in Figure <ref> demonstrates the robot's generalization ability to complete the collecting objects task with new objects.






 §.§ Performance of the robot in simulation: new scenario - new objects


We also evaluate the robot's performance in a completely new scenario using new objects. The experiment involves setting up a drink tray with various items, testing the robot's ability to adapt to a new task and handle unfamiliar objects. For this task, we utilized the stacking object policy and adjusted the subtask trajectory end poses to accomplish the task successfully.
Figure <ref> shows the sequence of snapshots detailing the robot's performance in setting up a drink tray. This image highlights the robot's capacity to generalize from its training and successfully perform tasks in new scenarios with new objects.






































 §.§ Perception of our real robot experiments

To discuss our real robot experiments, we provide an example of our experiment setup for the collecting bottles task (Fig. <ref>), for the stacking objects task (Fig. <ref>), and for the hammering task (Fig. <ref>). The robot's perception system is capable of identifying and localizing objects in its environment, which enables it to execute the task accurately.

The experiment demonstrates the integration of perception and action, where the robot uses its sensory inputs to detect various objects to guide its movements and complete the task of bottle collection.










 §.§ Failure cases during real robot experiments


During the real robot experiments, several failure cases were observed that can be attributed to the differences between simulation and real-world conditions (See Fig. <ref>). These failures provide valuable insights into the challenges of transferring learned policies from simulation to real robots. The main factors contributing to these failures include:

Gap between simulation and real robot: The objects used during the experiments in the real world differ in shape, size, weight, and material properties compared to those used in the simulation. These discrepancies can affect the robot's ability to accurately predict and execute the required trajectories, leading to unsuccessful task completion.


Inverse kinematics solver and command frequency: The inverse kinematics (IK) solver and the frequency of sending commands to the robot differ between the simulation and the real robot. In the simulation, the IK solver and command frequency are typically more idealized and may operate at higher rates, leading to smoother motions. In contrast, the real robot's controller might introduce slight delays due to various factors (e.g., network loads), causing deviations in the robot's movement and impacting task execution.

Trajectory prediction errors: Inaccuracies in the predicted trajectory can lead to the robot's end-effector being misaligned with the object, preventing successful grasping and manipulation. Poor grasping or inadequate grip strength can cause objects to slip during manipulation. 

Execution of actions: Using an open-loop controller during the execution of actions means that the robot follows the predicted trajectories blindly without feedback corrections. In real-world scenarios, this lack of feedback can lead to accumulated errors and deviations from the intended path, causing failures in task completion. In such an open-loop scenario, accurate calibration between the robot's camera and its manipulator is crucial and calibration errors can result in incorrect positioning and unsuccessful task execution.


These factors highlight the importance of addressing the sim-to-real gap and improving the robustness of robotic systems when transitioning from simulation to real-world applications. Future work should focus on integrating closed-loop feedback controllers to mitigate some of these issues and achieve more reliable performance in real-world scenarios.






 Paper Title: Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration.



















































1,3]Ngoc Son Nguyen


1,3]Son Van Nguyen


2,3]Tung Lecor1
lttung@fit.hcmus.edu.vn

[cor1]Corresponding author

[1]organization=Faculty of Mathematics and Computer Science, University of Science,
            
            city=
Ho Chi Minh,
            
            
            country=Vietnam

[2]organization=Faculty of Information Technology, University of Science,
            
            city=Ho Chi Minh,
            
            
            country=Vietnam

[3]organization=Vietnam National University,
            
            city=Ho Chi Minh,
            
            
            country=Vietnam







§ ABSTRACT 
 Visual Question Answering (VQA) has recently emerged as a potential research domain, captivating the interest of many in the field of artificial intelligence and computer vision. Despite the prevalence of approaches in English, there is a notable lack of systems specifically developed for certain languages, particularly Vietnamese. This study aims to bridge this gap by conducting comprehensive experiments on the Vietnamese Visual Question Answering (ViVQA) dataset, demonstrating the effectiveness of our proposed model. In response to community interest, we have developed a model that enhances image representation capabilities, thereby improving overall performance in the ViVQA system. Specifically, our model integrates the Bootstrapping Language-Image Pre-training with frozen unimodal models (BLIP-2) and the convolutional neural network EfficientNet to extract and process both local and global features from images. This integration leverages the strengths of transformer-based architectures for capturing comprehensive contextual information and convolutional networks for detailed local features. By freezing the parameters of these pre-trained models, we significantly reduce the computational cost and training time, while maintaining high performance. This approach significantly improves image representation and enhances the performance of existing VQA systems. We then leverage a multi-modal fusion module based on a general-purpose multi-modal foundation model (BEiT-3) to fuse the information between visual and textual features. Our experimental findings demonstrate that our model surpasses competing baselines, achieving promising performance. This is particularly evident in its accuracy of 71.04% on the test set of the ViVQA dataset, marking a significant advancement in our research area. The code is available at <https://github.com/nngocson2002/ViVQA>.


















Visual Question Answering ViVQA EfficientNet BLIP-2 Convolutional.














§ INTRODUCTION


The rapid enhancement of multi-modal processing in recent years <cit.> has not only opened up vast potential but has also considered a range of remarkable challenges. Among these, the field of VQA emerges as a critical problem in comprehending multi-modalities. This task entails empowering computers to interpret and respond to queries based on the content of images, necessitating a synergistic integration of advancements in both image processing and natural language processing. Despite its complexities, VQA is an essential and challenging task in the ongoing evolution of multimedia data. It represents a significant step towards understanding and integrating diverse forms of information, a cornerstone in the development of more intelligent and versatile Artificial Intelligence (AI) systems.







However, a notable challenge is that the majority of datasets used to train current VQA models are in English. Most research efforts in this field have prioritized languages with abundant resources, leaving low-resource languages like Vietnamese largely overlooked. This presents significant issues regarding the applicability of VQA models to questions written in Vietnamese and their ability to comprehend Vietnamese culture. In this context, focusing on the ViVQA task is invaluable, as it plays an important role in driving the development of innovative approaches in this specific domain. To address this gap, an increasing number of VQA models for Vietnamese have been established, such as those proposed in recent studies <cit.>, significantly contributing to this field. Our work is also no exception, aiming to tackle the ViVQA challenge by developing a novel system for ViVQA that leverages models based on the Transformer architecture <cit.>, which have been pre-trained on various tasks related to both vision and language, such as UNITER  <cit.>, CLIP <cit.>, BLIP <cit.>, and Flamingo <cit.>.



A fundamental aspect of VQA systems is the effective extraction and representation of textual and visual features in meaningful ways. The development of pre-trained models in text and image processing plays a crucial role in this aspect. However, it is essential to recognize that each type of model brings unique strengths to the table and should be evaluated from diverse perspectives. In our model, multi-modal features are intricately connected and integrated through a specialized fusion module. Building on the successes of previous models, our approach focuses on refining the semantic representations of uni-modal.


Although the Transformer architecture is renowned for its ability to analyze complete images, it sometimes overlooks crucial small details. This oversight is a significant challenge in VQA, as these details often contain vital information for understanding and answering questions. Shen et al. <cit.> contended that a VQA model relied on the Transformer architecture, exclusively considering global dependencies, is unable to effectively capture image context information. Through studies, they also proved that using both local and global features in image processing improves performance. Clearly, it is essential to consider the contextual representation of images from both local and global perspectives. Therefore, with this consideration in mind, we propose an innovative integration of transformer-based and convolutional methods in image processing to enhance the feature extraction module in VQA. This integration aims to enhance the visual information by combining the detailed local features extracted by the Convolutional Neural Network (CNN) with the broad global features derived from Vision Transformer-based models. Our novel component has the potential to evolve alongside advancements in visual feature extraction models by incorporating current improvements into our sophisticated combination. The promising results on the Vietnamese VQA dataset demonstrate that our proposed approach is more efficient than existing competitive baselines and plays an important role in representing images in VQA processing.

This paper makes several pivotal contributions to the field of ViVQA, highlighted as follows:

    
  * We propose an innovative methodological advancement by integrating transformer-based and convolutional methods in image processing. Our novel representation consists of local and global consideration through the content of images.
    
  * Our novel visual feature extraction is integrated into the ViVQA approach, employing the latest fusion module to deploy the efficient system.
    
  * Through the detailed experimental analysis and comparative results, our  method demonstrates the potential performance over existing competitive baselines. This achievement is a significant stride in leveraging multi-view feature representation, enhancing the understanding of different modalities within ViVQA. 



§ RELATED WORKS



 §.§ Overview

VQA methods typically consist of three primary components: modules for feature extraction from textual and visual inputs, a multi-modal fusion module, and a classifier or generator module. Initially, Antol et al. <cit.> introduced the VQA task alongside a simple baseline, utilizing VGGNet <cit.> and Long Short-Term Memory (LSTM) <cit.> for visual and textual feature extraction, respectively. They then fused these features into a joint representation for classifying candidate answers. Subsequently, Yuling Xi et al. <cit.> introduced an approach focusing on object relationships within images to address VQA tasks. Their method utilized a question-based attention mechanism to highlight relevant object regions, enabling accurate responses to questions regarding image content. Additionally, Yirui Wu et al. <cit.> proposed a baseline method emphasizing both global and local object relationships, underscoring the significance of comprehensive relation reasoning. Their approach leveraged information from the entire image for global understanding while the local aspect focused on modeling relationships among multiple objects to facilitate answer generation. Notably, these baselines all share the commonality of employing pre-trained CNN models like ResNet <cit.>, VGGNet <cit.>, and FasterRCNN <cit.> for image feature extraction. In terms of text embedding, pre-trained word embeddings such as GloVe <cit.> or Word2Vec <cit.> are often used along with LSTM <cit.> or Gated Recurrent Unit (GRU) <cit.> networks to extract linguistic features. In recent years, researchers have increasingly turned to Transformer-based models such as ViLBERT <cit.>, X-LXMERT <cit.>, and VLMo <cit.>. Consequently, later works often rely on these models, leveraging the capabilities of pre-trained models.

The primary focus of this paper is the examination of monolingual ViVQA. In previous research, Tran et al. <cit.> introduced a model that utilized Hierarchical Co-Attention <cit.>, establishing the first baseline for ViVQA dataset. Their experiments demonstrated that their system outperformed two baseline models, LSTM <cit.> and BiLSTM, on the ViQA dataset. Following this, with the advent of transformers, Nguyen-Tran et al. <cit.> proposed an approach that incorporates a Bi-directional Cross-attention architecture. This model was designed to more effectively comprehend the complex interplay between visual and linguistic features, specifically in the Vietnamese context. Another notable contribution by Nguyen et al. <cit.> was the introduction of Parallel Attention Transformer (PAT), a novel multimodal learning scheme. This method effectively integrates the advantages of grammar and contextual understanding in Vietnamese. More recently, Tran et al. <cit.> proposed leveraging BEiT-3 <cit.> as a fusion encoder to intricately model the deep interactions between images and questions. Furthermore, they employed BARTpho <cit.>, a monolingual sequence-to-sequence model pre-trained for Vietnamese, to enhance question representations. Remarkably, this model has achieved the current state-of-the-art performance on the ViVQA dataset. This research significantly advances the field of VQA, particularly in the Vietnamese language context, showcasing the effectiveness of novel models in capturing the nuanced interactions between visual and linguistic elements.




 §.§ ViVQA Models in Previous Studies

In this section, we will briefly present two methods that currently achieve the best performance on the ViVQA task and point out their weaknesses, which motivates our study.


  §.§.§ Multimodal Contextual Attention Model

In 2022, Anh Duc Nguyen et al. <cit.> introduced the Multi-vision Contextual Attention (MCA) model, presenting an innovative approach to image feature extraction that seamlessly integrates both global and local features. This model excels in learning attentional embeddings, capturing diverse contexts from both images and questions while being mutually guided.



As depicted in Figure <ref>, the image initially extracts local features corresponding to specific regions or parts of objects in the image, acquired through the utilization of ResNet <cit.>, a convolutional neural network. Simultaneously, global features encapsulate fundamental information about the contextual surroundings within the image, including associations among objects, and are derived using the Vision Transformer (ViT) <cit.>. For question embedding, PhoBERT <cit.> is employed to extract features from questions. Based on the obtained visual and textual features, a multi-branch contextual attention fusion hierarchically integrates visual and textual attention features, guiding each other and fusing them into a multimodal feature representation. This pioneering model represents a significant advancement in feature extraction, showcasing the integration of diverse contextual elements for a more comprehensive understanding of multimodal data.

Despite the combination of CNN and a Transformer in this model, it's worth noting that ResNet is not the latest CNN model. In our proposed architecture, we aim to innovate by combining both local and global features inspired by the MCA model. Through this approach, we anticipate achieving better performance.


  §.§.§ BARTPhoBEiT

Tran et al. <cit.> have recently introduced an innovative methodology that integrates pre-trained sequence-to-sequence and image Transformers customized for the Vietnamese language. This pioneering approach draws inspiration from BEiT-3 <cit.>, serving as the fundamental framework for their model.



As shown in Figure <ref>, the image embedding module utilizes BEiT v2 <cit.> to extract global features from images, while for question embedding, BARTPho <cit.> is employed to extract question features. Subsequently, the visual and question features are fused using a Multiway Transformer architecture <cit.>. These Transformers dynamically route information to the appropriate expert for each modality, ensuring the capture of modality-specific details. This unique feature significantly enhances the model's efficacy in handling diverse data modalities, emphasizing its versatility in cross-modal applications. Furthermore, the robustness of this model relies on the Transformer architecture, enabling seamless integration of an extensive volume of pre-trained data across diverse domains, resulting in state-of-the-art performance on the ViVQA dataset.

Despite the commendable results of their work, we have identified the complexity of image processing and the absence of information about local features as areas for further exploration. Building on the insights from their study, we are motivated to focus on the development of methods that enhance the performance of image processing, particularly for the ViVQA task.



 §.§ Dataset for ViVQA

In the context of Vietnamese, the development of data for VQA is not as advanced as it is in English. The available resources are limited, presenting a scarcity of data specifically tailored for ViVQA. This dearth underscores the challenges and gaps in the field, as only a modest amount of data is currently accessible for research and improvement in ViVQA. It is challenging to create models that can effectively generalize to new inputs when there is insufficient data to capture the entire variety of text and images. Furthermore, a significant element influencing the models' performance is the dataset's quality. Predictions might be erroneous and ineffective generalization can result from incomplete and low-quality data. Within the scope of this study, our primary focus is on monolingual Vietnamese datasets. We will survey datasets that align with our research objectives to serve as the foundation for our model.


  §.§.§ OpenViVQA Dataset

In 2023, Nghia et al. <cit.> introduced the OpenViVQA dataset <cit.> aims to showcase the inherently open-ended nature of both questions and answers. In order to seamlessly align with the intricacies of the Vietnamese language, the OpenViVQA dataset thoughtfully incorporates a diverse array of images captured within the rich tapestry of Vietnam's landscapes. This deliberate inclusion not only serves to showcase the linguistic nuances but also aims to highlight the myriad characteristics that distinctly define Vietnam in comparison to other regions. By strategically curating scenes from Vietnam, the dataset offers a comprehensive representation of the country's unique cultural, environmental, and visual elements, contributing to a nuanced understanding within the realm of VQA research.

The dataset is divided into two distinct components: “Text QA" and “Non-text QA". Specifically, the non-text QAs are designed to extract information pertinent to objects, encompassing their attributes and relational aspects. On the contrary, it is categorized as Text QA if it effectively leverages textual information emanating from the images themselves or employs the words present within the images as a means to pose questions. As a result of the inclusion of Text QA within the dataset, consequently making Optical Character Recognition (OCR) <cit.> a more involved process, our primary focus does not lie in this domain. Our paramount focus is intricately tied to the detailed examination of large objects depicted in the images. Thus, the OpenViVQA dataset does not align with our preference for this particular study.








  §.§.§ ViVQA Dataset
 
The inaugural dataset in ViVQA, crafted by Tran et al. <cit.> in 2021, stands as a pioneering contribution to the field. Drawing inspiration from established English VQA benchmarks and COCO-QA <cit.>, this dataset has evolved into a widely embraced benchmark for related research endeavors. The creation process involved leveraging machine translation for the conversion of questions and answers, followed by meticulous manual validation to ensure correctness and linguistic fluency in the translated content. 

The ViVQA dataset <cit.> encompasses a collection of 10,328 images paired with 15,000 corresponding questions, intricately tailored to reflect the content within the images. To facilitate experimentation and evaluation, the dataset underwent a random partitioning into training and test sets, maintaining an 80:20 ratio. Noteworthy is the dataset's categorization of questions into four distinct types: Object, Number, Color, and Location, constituting 41.55%, 14.81%, 20.82%, and 22.82% of the total questions, respectively. 

This dataset is particularly pertinent to our research objectives as our paper predominantly delves into advanced image processing techniques. Our primary focus lies in extracting detailed information related to objects, encompassing their attributes, relational aspects, and the overall landscape depicted across the image. Consequently, we have deliberately chosen this dataset for our study, opting against OpenViVQA. This decision is attributed to the existence of questions related to textual content within images in OpenViVQA, which does not align with the specific aspects of image analysis and processing we are exploring in our research.















































































































§ METHODOLOGY

In this paper, we approach the ViVQA task as a classification problem. Specifically, given an image I and a question Q, the goal is to determine the most probable answer â from a predefined set of answer options A:

    â = max_a ∈ A P(a|I,Q,A)


In this section, we present the detail of our ViVQA model architecture, illustrated in Figure <ref>, drawing inspiration from BARTPhoBEiT <cit.> with specific modifications in the visual processing aspect. We also provide detailed descriptions of the main image processing steps employed in our model. Our proposal advocates for the implementation of visual processing with frozen parameters, effectively reducing both the model parameters and training costs.





 §.§ Image Embedding Module
 
Utilizing the capabilities of pre-trained models, our image embedding module proves to be both effective and powerful, capturing valuable features through pretraining tasks. We employ both BLIP-2 <cit.> and EfficientNet <cit.> to extract visual features from both local and global respectively, amalgamating them to derive the final representation for the image. A distinctive feature of our module is its separation from the overall architecture, ensuring that its parameters remain static throughout the training process. This design choice significantly reduces the overall number of parameters in our model and shortens the training duration.



  §.§.§ BLIP-2

The first model in the BLIP family, Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (BLIP), was proposed by Li et al. <cit.>. BLIP introduced a novel Vision-Language Pre-training (VLP) framework that supports a wider range of downstream tasks than existing methods. This is achieved by jointly training models on extensive language and image datasets, thereby leveraging the complementary strengths of both modalities. 

Following this, Li et al. <cit.> proposed BLIP-2, an enhanced version of the original model with several adjustments in both the training strategy and the architecture. BLIP-2 introduces a pioneering method that leverages the capabilities of pre-trained vision models and language models. Unlike BLIP, which involves end-to-end training and often incurs high computational costs and lengthy training durations, BLIP-2 opts to freeze these pre-trained models during the training process. This strategy results in improved performance at a reduced computational cost and addresses the issue of catastrophic forgetting. Moreover, it ensures that the model capitalizes on the rich representations learned by the visual and language model, thereby enhancing its overall performance and effectiveness in comprehending both image and text inputs. With these strengths of BLIP-2, we decided to adopt it to take advantage of global image-level feature extraction.

The central aim of BLIP-2 is to facilitate mutual understanding between the image and language. To achieve this objective, the Querying Transformer (Q-Former) module is specifically engineered to bridge the gap between vision and language. Following the Figure <ref>, the Q-Former architecture in BLIP-2 incorporates (left) an image transformer utilizing an encoder-only transformer and (right) a text transformer, capable of functioning as both a text encoder and a text decoder. Both the image transformer and text transformer share the same self-attention layers. Given our focus on obtaining image representations for the ViVQA system, we are particularly interested in the image transformer within the Q-Former, which returns the visual features. A specific number of learnable query embeddings serve as input for the image transformer. The queries, each comprising 32 learned query tokens with a dimension of 768, contextualize the query tokens through the cross-attention mechanism with the representations of the image patches encoded by a large (frozen) ViT <cit.>. The visual tokens serve as the output of the Q-Former.



Notably, a crucial stage involves representation learning, with the primary aim of pre-training within BLIP-2 being to refine the Q-Former's ability to enable queries to extract visual representations that are maximally informative regarding the accompanying text. Consequently, the features of the image generated by BLIP-2 encompass an ample amount of information, particularly in terms of global aspects, satisfying our ViVQA system's needs. The output query representation in ℝ^32 × 768, significantly smaller than the size of the frozen image features (e.g., 257 × 1024 for ViT-L/14). Thus, the image I representation after BLIP-2 feature extraction can be formulated as:

    V_G = BLIP-2(I) ∈ℝ^32 × 768




  §.§.§ EfficientNet

We integrate the EfficientNet architecture introduced by Tan et al. <cit.>, a sophisticated CNN, for local image-level feature extraction. Intuitively, CNN architectures inherently utilize kernels sliding over regions of the image, effectively capturing information locally. Previous CNN architectures commonly face a trade-off between model complexity and performance, wherein as performance improves, model complexity increases. This trade-off involves scaling, which can occur through depth scaling, width scaling, and resolution scaling.

Depth scaling entails increasing the depth of a neural network, typically resulting in improved performance. Width scaling involves increasing the number of channels (or neurons) in each layer, thereby enhancing the model's capacity to capture a broader range of features at each level and improving its representational power. Resolution scaling, meanwhile, requires resizing input images to higher resolutions, potentially leading to better performance as higher-resolution images contain more detailed information. 

EfficientNet <cit.> is an improvement version basing on CNN that implements the concept of compound scaling, which enhances performance by uniformly scaling all dimensions of depth, width, and resolution. Its power is in how these three dimensions are balanced and adjusted using a systematic methodology. The model employs Mobile Inverted Bottleneck (MBConv) <cit.> layers, combining depth-wise separable convolutions and inverted residual blocks. This bottleneck design enables efficient learning while preserving a high degree of representational power. The model's potency has been demonstrated through experiments by Tan et al. <cit.>, which showed that models in the EfficientNet family achieve performance equivalent to other models while maintaining low complexity. To leverage these capabilities, we employ an EfficientNet-B7 model, a pre-trained model based on adversarial training <cit.>. It is initialized with pre-trained weights on ImageNet <cit.>, providing a robust foundation for directly extracting features for our ViVQA system.

The input image I, denoted as ℝ^C × H × W, where C,H and W stand for channels, height, and width respectively, undergoes processing within the pre-trained EfficientNet model. The features are extracted from the output of the final head before undergoing average pooling and a fully connected layer. This process generates a representation capturing the local features of the image, expressed as ℝ^2560 ×H/32× W/32. It is noteworthy that our original image data is standardized to the dimensions of ℝ^3 × 224 × 224. Consequently, the image representation following EfficientNet feature extraction has dimensions ℝ^2560 × 7 × 7.

    V_L = EfficientNet(I) ∈ℝ^2560 × 7 × 7


This approach allows us to capture intricate local features from the images, providing a rich and informative representation essential for subsequent stages in our model architecture. 



  §.§.§ Image Embedding

In this stage, we will amalgamate the representations of BLIP-2 and EfficientNet to derive the final output of the image embedding module. Initially, an inconsistency was observed in the dimensions of BLIP-2 (<ref>) and EfficientNet (<ref>). Thus, we will implement several steps to transform the representation of EfficientNet to align it with the output of BLIP-2. This transformation is crucial to facilitate operations for seamlessly combining the local and global features.

The transformation involves employing Adaptive Average Pooling to downsample from ℝ^2560 × 7 × 7 to ℝ^2560 × 1 × 32:

    V_L = AdaptiveAveragePooling(V_L) ∈ℝ^2560 × 1 × 32


Subsequently, its dimensions are permuted to ℝ^32 × 1 × 2560, and another round of Adaptive Average Pooling is applied for downsampling to ℝ^32 × 1 × 768:

    V_L = Permute(V_L) ∈ℝ^32 × 1 × 2560


    V_L = AdaptiveAveragePooling(V_L) ∈ℝ^32 × 1 × 768


Through the flattening process, we derive the ultimate output of the image embedding module:

    V_L = Flatten(V_L) ∈ℝ^32 × 768


Finally, the fusion of local features and global features is computed as follows:


    V = ℱ(V_G, V_L) ∈ℝ^k × 768

where k relies on the operation ℱ. This equation represents the calculation for combining the global features, denoted as V_G, and the local features, denoted as V_L, through a fusion function ℱ.



 §.§ Text Embedding Module
 

Throughout Tran et al.'s study <cit.>, they observed the prevalence of grammatical errors and translation inaccuracies in ViVQA questions, even when validated by highly qualified annotators. Therefore, instead of relying on PhoBERT <cit.> to extract question information, which predominantly comprehends word meanings in diverse contexts, our preference is to employ BARTpho <cit.>. BARTpho stands out as a denoising autoencoder specifically crafted for pretraining sequence-to-sequence models geared towards Vietnamese language processing. The training regimen involves a two-step process: (1) corrupting the input text using a designated noising function and (2) learning to reconstruct the original text. This pretraining task empowers BARTpho with the capability to acquire robust representations resilient to noise, ambiguity, and variations in the input text. Hence, our approach leans towards leveraging the capabilities of BARTpho.

BARTpho offers two distinct variants, specifically BARTpho_syllyble, which is optimized for processing at the syllable level, and BARTpho_word, which is designed for handling language at the word level. In the case of BARTPho_word, the sentence must undergo preprocessing, being appropriately split into individual words and phrases. Subsequently, the input of complete sentences is fed into both the encoder and decoder, with the top hidden state of the decoder serving as a representation for each word. In our model, we use BARTpho_word and the textual features T can be articulated as:

    Q = BARTpho_word(T) ∈ℝ^(L+2) × 1024

where L + 2 with L is the max sequence length, adding 2 special token [CLS] and [SEP] representing the start-of-sentence and the end-of-sentence token.

Since the BLIP-2 in the image embedding module is predominantly trained on English data (specifically, images with English captions) and the text embedding module is tailored for Vietnamese, we set the parameters of the text embedding module to be trainable. This adaptability enables the text representation to dynamically adjust and align with the visual representation within the image embedding module.





 §.§ Multi-modal Fusion Module

Before forwarding the representations of visual and question features to BEiT-3 for multimodal fusion, a Feed-Forward Network (FFN) layer is applied on the textual features. The purpose is to diminish the dimensionality from 1024 (<ref>) to 768. This reduction is imperative due to the requirement of concatenating the embeddings of a given question and an image within this component. The dimension of the question is now given by:

    Q = FFN(Q) ∈ℝ^(L+2) × 768


This preprocessing step optimizes the dimensionality of the textual features, enhancing their compatibility for subsequent fusion with visual features. Subsequently, the visual (<ref>) and question (<ref>) features are concatenated:

    f_VQ = concatenate(V, Q) ∈ℝ^(k+L+2) × 768


Following this, the concatenated features are inputted into Multiway Transformers <cit.> to jointly encode the image-question pair. Each Multiway Transformer block is composed of a shared self-attention module and a set of feed-forward networks tailored for diverse modalities. Each layer includes both a vision expert and a language expert, each responsible for processing the input tokens directed to them. Leveraging a group of modality experts incentivizes the model to capture a broader range of modality-specific information. The shared self-attention module acquires the ability to discern alignments between diverse modalities, facilitating profound fusion for tasks involving multiple modalities, such as vision-language tasks.

The result will have the same dimension as the input f_VQ (<ref>). The first vector in the output fusion is earmarked for classification, denoted as f̂_VQ∈ℝ^1 × 768.
Prior to being input into the Classifier module, f̂_VQ undergoes processing through a pooler layer, involving the normalization, a FFN layer with hidden and output dimension of 768, and application of the hyperbolic tangent function as depicted below:

    f̂_VQ = Norm(f̂_VQ)


    f̂_VQ = FFN(f̂_VQ)


    f̂_VQ = Tanh(f̂_VQ)




 §.§ Classifier Module

We integrate a classification head to predict the answer, consisting of the following components: (1) a FFN layer with hidden and output dimensions of 768 and 1536, (2) a normalization layer, (3) the GELU activation function, and (4) an additional FFN layer that maps the input dimension of 1536 to the number of classes (353 candidate answers).

    l = FFN(f̂_VQ)


    l = Norm(l)


    l = GELU(l)


    l = FFN(l)


Following this, a softmax function is applied to convert the output into a distribution within the range of 0 to 1:

    p̂ = softmax(l)

The candidate with the highest probability is then chosen as the proper answer.


§ EXPERIMENTS

In this section, we conduct evaluations of our model using the ViVQA dataset <cit.>. Additionally, we compare our model's performance with that of previous ViVQA models that have achieved state-of-the-art results on the same dataset.

By formulating the VQA task as classification, we employ Cross-Entropy Loss as the loss function and assess our model's performance on the ViVQA dataset, utilizing accuracy as the main metric.



 §.§ Dataset

As we mentioned in section <ref>, the ViVQA dataset is the subject of our investigations. A training set of 11,999 Q-A pairs and a test set of 3,001 Q-A pairs comprise the dataset. The detail of the dataset is shown in Table <ref>.





 §.§ Evaluation Metrics

In formulating the ViVQA as a classification problem, we assess the model's effectiveness using key metrics: F1 score, Precision, Recall, and Accuracy. In alignment with our objective to compare against the baseline in our experiment, we prioritize accuracy over other metrics. This choice stems from the fact that all baseline models are evaluated based on accuracy, with some not utilizing additional metrics. Thus, in our experimental endeavor to evaluate the influencing factors of the model, accuracy will serve as the primary metric for a more straightforward analysis.


  §.§.§ Accuracy

Accuracy or exact match is defined as the ratio of the number of correct predictions to the total number of ground truth answers. Let N represent the total number of ground truth answers. The accuracy is calculated as follows:

    Acuracy = 1/N∑_i=1^N a_i


    a_i = 
            
          1,    if ŷ_̂î = y_i 
    
          0,    otherwise

where ŷ_̂î is the predicted answer, and y_i is the ground truth answer for question i.








  §.§.§ Recall

This metric, relying on tokenized versions, measures the proportion of correctly identified tokens in the ground truth answer that are also present in the predicted answer. Let P_i represent the tokens in the predicted answer and GT_i denote the tokens in the ground truth answer for the question i-th. The recall for the question i-th is determined as follows:

    r_i = P_i ∩ GT_i/GT_i

Let N represent the total number of ground truth answers, so the overall recall is computed as:

    Recall = 1/N∑_i=1^N r_i



  §.§.§ Precision

Precision, in the context of tokenized forms, evaluates how many of the predicted tokens match those in the ground truth answer. Let P_i represent the tokens in the predicted answer and GT_i denote the tokens in the ground truth answer for the question i-th. The precision for the question i-th is calculated as follows:

    p_i = P_i ∩ GT_i/P_i

Let N represent the total number of ground truth answers, so the overall recall is computed as:

    Precision = 1/N∑_i=1^N p_i



  §.§.§ F1 score

The F1 score is essentially computed from both recall (<ref>) and precision (<ref>) of each individual question. The F1 score for the question i-th is calculated as follows:




    f_i = 
            
          0,    if p_i = 0, r_i = 0 
    2 × p_i × r_i/p_i + r_i,    otherwise

Let N represent the total number of ground truth answers, so the overall F1 score is computed as:

    F1 = 1/N∑_i=1^N f_i




 §.§ Experimental Settings

In our experiments, the hyperparameters presented in Table <ref> were utilized. The training process spanned 20 epochs with a batch size of 65, a drop path rate of 0.3, a learning rate of 3e-5, weight decay set at 0.01, and a warmup ratio of 0.1. The input image resolution was 224 x 224. We employed AdamW <cit.> as the optimizer for updating parameters. The model training took place on a computing system equipped with NVIDIA TESLA P100-16GB GPUs and 32GB of RAM.



Remarkably, within the multi-modal fusion module, a crucial hyperparameter during training is the drop path rate, also known as stochastic depth <cit.>. The role of drop path is akin to dropout <cit.>, a regularization technique employed to counteract overfitting in neural networks. Both methods introduce randomness during training, yet they operate at different levels of network architecture. Drop path randomly omits individual samples within a batch during training, thereby impacting the depth of the network's computational graph. In contrast, dropout randomly eliminates units within layers, thereby altering the connectivity between layers. Drop path selectively bypasses specific connections within the neural network for each sample independently, while dropout affects all units within a layer. By randomly omitting paths (connections) during training, drop path encourages the network to acquire more robust features by discouraging reliance on specific paths. This diversification of learned features aids in mitigating overfitting by facilitating a broader exploration of the network's parameter space.


  §.§.§ Training Strategy

In our paper, we adopted a systematic approach to training our model to ensure optimal performance. Initially, we partitioned our dataset into a training set and a validation set in an 80:20 split. This division allowed us to fine-tune and determine the most effective parameters for our model, with a particular emphasis on identifying the most suitable hyperparameters, including the drop path rate, the number of encoder layers, and the count of encoder attention heads. For the rest of the hyperparameters, we closely followed the training settings used for fine-tuning the BEiT-3 model <cit.> on the VQAv2 dataset <cit.>.

To further refine our training process and ascertain the ideal number of training epochs necessary for model convergence, we employed a 5-fold cross-validation strategy on the training set. This approach enabled us to determine that our model reached convergence after an average of 20 epochs across all folds. Armed with this insight, we proceeded to train our model on the complete training set for the identified duration of 20 epochs.

After selecting the optimal hyperparameters, we tested each model ten times using ten different random seeds. This approach allowed us to determine the statistical significance between any two models using a two-sample t-test.



 §.§ Experimental Results



  §.§.§ Impact of Fusion Operation Types on the model

In the initial phase of our study, we conducted an exploration to identify the optimal fusion function ℱ, as defined in equation (<ref>), for our model. This investigation encompassed various strategies, including element-wise multiplication, element-wise addition, and concatenation, applied to integrate both global and local image features. Despite the simplicity of the fusion strategies employed, they demonstrated noteworthy effectiveness in enhancing the overall performance of our model.

Table <ref> presents the outcomes of our experimentation. For the element-wise operations of multiplication and addition, we have features of visual ∈ℝ^32 × 768, and for concatenation, we have features in ℝ^64 × 768. It is evident that the concatenation operation outperforms the other operations, achieving the highest level of performance. Consequently, we have chosen concatenation as the fusion function ℱ for subsequent experiments.

It is noteworthy that the outcome of the multiplication operation is markedly lower compared to concatenation and addition. Our investigation revealed that the diminished performance can be attributed to the multiplication of two features (local ∈ℝ^32 × 768 and global ∈ℝ^32 × 768) resulting in a new matrix with dimensions ∈ℝ^32 × 768 that exhibits a high degree of sparsity. The boxplots in Figure <ref> indicate that, on average, the elements in the matrices after addition and concatenation operations demonstrate elevated levels and substantial spreads. In contrast, the multiplication operation yields a sparse matrix characterized by elements that tend to approach zero closely, displaying a narrow range of spreads. This pattern suggests that the sparse matrix resulting from multiplication leads to information loss in the visual representation, thereby accounting for its diminished performance.







  §.§.§ Impact of CNNs on the model

In the subsequent phase, we conducted a meticulous investigation into the influence of various CNN architectures on the efficacy of our model. The choice of a CNN architecture plays a pivotal role in shaping our model's performance. We comprehensively evaluated diverse CNN models, including ResNet <cit.> and VGGNet <cit.>, alongside the EfficientNet family architecture. This assessment was rooted in accuracy metrics and an examination of the parameter count for each model.

Table <ref> presents a synthesis of accuracy and parameter count data for each CNN model. The findings reveal that the EfficientNet family not only surpasses other models in terms of accuracy but also, with a moderate parameter count, exhibits superior performance compared to the conventional ResNet and VGG architectures. As observed, VGG achieves an accuracy of over 70%, but it requires hundreds of millions of parameters. In contrast, EfficientNet achieves comparable accuracy with only tens of millions of parameters. We selected the large-scale model (EfficientNet-B7) as the optimal choice, offering elevated accuracy with reasonable model complexity for our model.





  §.§.§ Comparison with other Methods

Due to the limited research on VQA tasks in Vietnamese, we benchmarked our model against several existing approaches that have demonstrated promising results in this domain.  Specifically, we compared our model to strategies proposed by other methods. LSTM-based approaches <cit.> and pre-trained word embeddings (PhoW2Vec <cit.>) proved ineffective in handling the complexities of VQA. Conversely, methods based on transformers demonstrated excellent performance in addressing this challenging task.



The comparative results are presented in Table <ref>. Our model achieves a remarkable accuracy of 71.04% on the test set, surpassing all other methods by a significant margin. This represents a substantial improvement of 37.19% over the least accurate model (LSTM + PhoW2Vec) and a notable gain of 2.46% over the second-best approach (BARTPhoBEiT). These results unequivocally demonstrate the superiority of our proposed method in addressing ViVQA.

Overall, many previous baselines lack information about local features, with the exception of MCA. The effectiveness of incorporating both local and global features has proven to be crucial for the success of previous literature. We will conduct an ablation study in the next section to demonstrate their influence. Although MCA considers both local and global features in image extraction and achieves commendable accuracy, it utilizes ViT, which is trained solely on image datasets. In contrast, our method leverages BLIP-2, which is pre-trained on both vision and language data. This vision-language pre-training enables BLIP-2 to capture more comprehensive and contextually rich information than that obtained from ViT.

With promising results achieved on the ViVQA dataset, our model has demonstrated effectiveness in the VQA task, particularly in its ability to extract, represent, and comprehend the relationship between images and text. To ensure the model effectively understands Vietnamese, we propose leveraging not only a powerful pre-trained model like BARTPho but also integrating a multi-modal information synthesis module like BEiT-3 to create a joint representation containing information from both text and images. Additionally, our model not only relies on the existing knowledge encoded in the pre-trained BARTPho model for Vietnamese, but we also refine it to suit multi-modal tasks between Vietnamese images and texts, thereby establishing a foundation for similar multi-modal studies in Vietnamese.













 §.§ Ablation Study

We conducted an ablation study to thoroughly investigate the contributions of BLIP-2 and EfficientNet to the overall outcome. As depicted in Table <ref>, models relying solely on BLIP-2 or EfficientNet for extracting image features exhibited lower accuracy. Whilst combining both methods led to improved performance, surpassing the results achieved with a singular approach.


Furthermore, we systematically selected the number of encoder attention heads and encoder layers in the multi-modal fusion module. In our system, this module plays a pivotal role in the model’s performance for combining information from the visual and question modalities. Initially, we fixed the number of encoder layers at 6 and varied the number of encoder attention heads. The results, as illustrated in Figure <ref>, highlight the optimal performance of our model with 6 encoder attention heads.



Subsequently, maintaining 6 encoder attention heads constant, we adjusted the number of encoder layers. The outcomes, showcased in Figure <ref>, underscore that the optimal number of layers for the encoder is 6.



After obtaining the ultimate components of our model, we investigated the impact of freezing the visual extractor on model performance. Due to limited infrastructure in the experimental environment, we only used EfficientNet-B7 from our image embedding module to conduct the experiment. The results, depicted in Table <ref>, indicate that freezing the EfficientNet-B7 led to an accuracy of 71.04% within 20 epochs, whereas the visual extractor is not frozen, the accuracy is slightly lower at 70.44%, with a significantly longer training time of 935 seconds. From these findings, we conclude that freezing the visual extractor can enhance efficiency without compromising accuracy, as well as saving a lot of training time, rendering it a viable strategy for training deep learning models in similar contexts.






 §.§ Discussion



  §.§.§ Dataset Challenges

Despite the comprehensive validation of the dataset by professionals, certain samples exhibit ambiguity, grammatical issues in ViVQA questions, or a lack of relevance between questions and answers. This highlights the challenge posed by the dataset's overall low quality, a significant hurdle for VQA tasks in Vietnamese, primarily due to the scarcity of high-quality datasets. As depicted in Figure <ref>, we illustrate instances where the ground truth answers are inappropriate, unclear, or unrelated to the corresponding questions. Particularly, discrepancies arise, such as the misidentification of the bear's color as brown instead of black, or the color of the cows being described as brown while the ground truth indicates white. Moreover, descriptions indicating a sandwich topped with quality ingredients are contradicted by an irrelevant answer, `Dog'. Similarly, instances where the depicted scene clearly shows a man riding a horse but is misinterpreted as a `mountain', and a girl taking a selfie, potentially in a restroom, yet labeled as a `Cage', further underscore the poor quality of the dataset in this study. These findings highlight the poor quality of the ViVQA dataset in this study, presenting additional challenges for researchers interested in improving dataset quality.








  §.§.§ Experimental Analysis

To start, we present insightful examples to analyze the performance of our model. Through experimental analysis, strengths and weaknesses become readily apparent in practical scenarios. The specifics of these instances are illustrated in Figure <ref>. 

In Figure <ref>-(a), both BLIP-2 and EfficientNet made individual incorrect predictions, suggesting that each model lacks comprehensive information from the image. BLIP-2, emphasizing the global context of the image, may overlook local details, while EfficientNet may struggle to capture broader contextual information. 

In Figure <ref>-(b), the model solely relying on BLIP-2 for visual extraction erroneously predicted the color of the vase as white, likely due to BLIP-2's tendency to prioritize global features, where much of the image appears white. Consequently, EfficientNet provided a correct answer by considering local details and determining the color of the vase to be red. 

In Figure <ref>-(c), the model utilizing only EfficientNet for visual extraction predicted that the location where a woman is holding a sandwich is the kitchen. This is attributed to EfficientNet's capability to focus on intricate details such as the sandwich and the dashboard of a car, which might resemble oven buttons. Conversely, BLIP-2, taking into account the global context, accurately identified the location as a car. 

Through experimental evaluation, it became evident that our model, along with its constituent components BLIP-2 and EfficientNet, exhibits both strengths and weaknesses. BLIP-2 excels in capturing the global context of images but may overlook finer local details, while EfficientNet focuses on intricate local features but may struggle with broader contextual information. These insights highlight the importance of considering both global and local features in model development for visual understanding tasks.



Another aspect we wish to address is the ambiguity observed in the answers provided in the dataset utilized for evaluation, as well as the complexity of the linguistic aspects. In Figures <ref>-(a,b), ambiguity arises between the ground truth answer and the predicted answer. We believe that the predicted answer could potentially be correct regarding the location of the object in the image; specifically, the man and artwork could be situated in the bathroom. Furthermore, in Figure <ref>-(c), there is an excessive amount of redundant information contained in the question, making it difficult for our model to extract relevant information. Consequently, our model predicts the overall scene (Room) instead of a specific place (Class). This underscores the importance of addressing ambiguity and redundant information in question-answer datasets, as they can significantly impact model performance and interpretation. With a sufficiently robust dataset, we believe we could tackle this challenge, representing the motivation driving our future focus.



An additional noteworthy aspect we would like to address pertains to the distribution of incorrectly predicted question types by our model. As depicted in Figure <ref>, our model inaccurately predicted the question type `object' in approximately 43.6% of cases, which accounts for nearly half of all incorrect predictions. This observation suggests that our model may lack sufficient information regarding the specific details of objects within the image. Consequently, addressing this limitation stands as a motivating factor for our future work.





  §.§.§ Transfer Learning Considerations

Our architecture is capable of efficiently transferring knowledge from a source domain, where the model undergoes initial training, to a target domain characterized by limited data or a demand for rapid deployment. This capability is underpinned by the flexibility of our design, which facilitates the easy replacement and adaptation of the backbone components.

In the context of linguistic applications, while our primary focus has been on monolingual Vietnamese, the model architecture can be readily adapted to other monolingual languages with minimal modifications to the Text Embedding module presented in section <ref>. This adaptability ensures that the linguistic component of our model can handle a diverse array of languages, thereby broadening its applicability. Similarly, in the visual domain, the model's backbones presented in section <ref>  can be replaced with alternative architectures, providing the flexibility to optimize performance based on specific visual tasks or datasets. 

Moreover, the versatility of our model extends beyond typical visual-linguistic tasks to specialized domains, such as Medical Visual Question Answering (MedVQA). Although our current work primarily addresses everyday scenes and objects, the transition to the medical domain is facilitated by the consistent input format of text-image pairs. This consistency aligns seamlessly with our architecture, enabling the model to leverage pre-existing knowledge and adapt to the nuanced requirements of medical imagery and terminology.

The ability to transfer learning effectively across these varied domains underscores the robustness of our architecture. By allowing for the modular exchange of core components, such as the text and visual backbones, our model provides a scalable and adaptable solution that meets the demands of diverse applications. This not only enhances the model’s utility across different linguistic and visual tasks but also ensures that it can be efficiently repurposed for new, domain-specific challenges.


§ CONCLUSION AND FUTURE WORK

In this study, we introduced a ViVQA system with the key integration of convolutional and base-transformer methods for vision embedding. The convolutional methods capture local features, while the base-transformer methods address global features, resulting in a more comprehensive image representation. Our experiments demonstrated the effectiveness of these combined approaches in enhancing our architecture's performance. However, we also encountered challenges related to the quality of the ViVQA dataset. The dataset's low quality posed significant difficulties, as it included several incorrectly labeled samples, which affected the overall performance. Despite our model's correct predictions from a human-like perspective, these predictions were not reflected accurately in the performance evaluations due to the dataset's inaccuracies. Furthermore, our model's performance is limited when addressing open-ended questions. The current dataset only encompasses four types of questions, and questions that deviate from these predefined types are not handled effectively by our model.

Despite the promising results, we recognize the need to address several challenges in future work. Our plans include improving the quality of the ViVQA dataset by conducting thorough reviews to ensure the accuracy of both questions and answers. We also intend to employ data augmentation techniques to further enrich the dataset. Additionally, as analyzed, our model still faces difficulties with questions concerning objects. To overcome this limitation, we aim to enhance object information by incorporating object detection models into the visual extraction process.

Our objective is to significantly improve the overall performance of the ViVQA system and contribute to the advancement of more efficient and reliable natural language processing in Vietnamese. By addressing these challenges, we hope to develop a more robust and accurate ViVQA system capable of handling a wider range of queries and providing better support for Vietnamese language applications.


§ ACKNOWLEDGEMENT

This research is funded by University of Science, VNU-HCM, Vietnam under grant number CNTT 2024–12.












    





























elsarticle-num-names 















 Paper Title: DD-rPPGNet: De-interfering and Descriptive Feature Learning for Unsupervised rPPG Estimation.


 






How to Use the IEEEtran  Templates


DD-rPPGNet: De-interfering and Descriptive Feature Learning for Unsupervised rPPG Estimation



    Pei-Kai Huang, Tzu-Hsien Chen, Ya-Ting Chan, Kuan-Wen Chen, Chiou-Ting Hsu,  Senior Member, IEEE 


Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan


    August 5, 2024
========================================================================================================================================================================================



§ ABSTRACT 

Remote Photoplethysmography (rPPG) aims to measure physiological signals and Heart Rate (HR) from facial videos.
Recent unsupervised rPPG estimation methods have shown promising potential in estimating rPPG signals from facial regions without relying on ground truth rPPG signals.
However, these methods seem oblivious to interference existing in rPPG signals and still result in unsatisfactory performance. In this paper, we propose a novel De-interfered and Descriptive rPPG Estimation Network (DD-rPPGNet) to eliminate the interference within rPPG features for learning genuine rPPG signals.
First, we investigate the characteristics of local spatial-temporal similarities of interference and design a novel unsupervised model to estimate the interference.
Next, we propose an unsupervised de-interfered method to learn genuine rPPG signals with two stages. In the first stage, we estimate the initial rPPG signals by contrastive learning from both the training data and their augmented counterparts. 
In the second stage, we use the estimated interference features to derive de-interfered rPPG features and encourage the rPPG signals to be distinct from the interference. 
In addition, we propose an effective descriptive rPPG feature learning by developing a strong 3D Learnable Descriptive Convolution (3DLDC) to capture the subtle chrominance changes for enhancing rPPG estimation. 
Extensive experiments conducted on five rPPG benchmark datasets demonstrate that the proposed DD-rPPGNet outperforms previous unsupervised rPPG estimation methods and achieves competitive performances with state-of-the-art supervised rPPG methods.
   



Unsupervised rPPG estimation,  local spatial-temporal similarity, de-interfered rPPG estimation, learnable descriptive convolution, descriptive feature learning.





§ INTRODUCTION



  
 

Remote photoplethysmography (rPPG) aims to measure physiological signals without any skin contact <cit.> by analyzing the facial chrominance changes reflected on skin <cit.>, and is used to capture  the heart rate related information <cit.>.
Many deep learning-based methods <cit.>, through the adoption of supervised learning techniques, have achieved significant breakthroughs in rPPG estimation. 
These supervised rPPG estimation methods rely on paired facial videos and ground truth rPPG signals to learn the estimation of rPPG signals from facial videos. However, collection of paired facial videos and ground truth rPPG signals is very costly, and also their quality highly depends on the adopted contact sensors, the involved subjects, and the environmental settings. 
Therefore, existing datasets, in terms of their quantity and quality, remain insufficient for developing robust supervised rPPG estimation methods.



In contrast, unsupervised rPPG estimation methods <cit.> have been developed to estimate rPPG signals directly from face videos without referring to paired ground truth rPPG signals.
These methods are developed based on several assumptions about the nature of rPPG signals.
For example, in <cit.>, the authors adopted contrastive learning, under the observation of temporal similarity of rPPG signals and the typical range of  human heart rate (within  40 to 250 beats per minute (bpm)), to develop a unsupervised rPPG estimation method.
Next, the authors in <cit.> further included the observation of spatial similarity within each individual video and the observation of dissimilarity across different videos to refine the contrastive learning in unsupervised rPPG estimation.



Furthermore,  in <cit.>, the authors proposed to augment the training data by using dissimilar signal frequencies for enhancing unsupervised rPPG estimation.
In addition, the authors in <cit.> proposed a non-contrastive unsupervised rPPG estimation approach to learn concentrated energy between the heart rate ranges.
Although these unsupervised rPPG estimation methods may perform well on datasets with minor interference, they usually fail to estimate accurate rPPG signals in datasets containing challenging interference.
Since facial videos are inevitably corrupted by various interference, such as head motions <cit.>, facial expressions <cit.>, video compression artifacts <cit.>, and lighting variations <cit.>, rPPG estimation is very sensitive to these interferencing sources. As shown in Figure <ref> (a), previous unsupervised rPPG estimation methods <cit.> often disregard the existence of interference in rPPG features and tend to extract interference-carrying rPPG signals. 
In Section <ref>, we will show that the rPPG signals estimated from existing unsupervised rPPG estimation methods, such as <cit.>, exhibit a notably positive correlation with the interference in non-facial background regions.
To eliminate interference in rPPG features, previous supervised methods <cit.> mostly relied on ground truth rPPG signals to model interference for learning genuine rPPG signals.
As to unsupervised rPPG estimation methods, which involve no paired ground truth rPPG signals, the task of extracting de-interfered rPPG signals is much more challenging compared to supervised methods.




In addition, estimation of rPPG signals is a nontrivial task because they are estimated from subtle chrominance changes reflected on skin. 
To capture these subtle skin color changes, previous supervised methods <cit.> included several pre-defined local descriptors into vanilla 3DCNN, such as 3D Temporal Difference Convolution (TDC) <cit.>, and 3D Central Difference Convolution (3DCDC) <cit.>, to learn descriptive features for enhancing rPPG estimation.
Notably, inclusion of learning descriptive rPPG features in unsupervised rPPG estimation remains unexplored.
Therefore, when dealing with facial videos corrupted by various interferences, rPPG estimation is indeed doubly challenging to unsupervised methods.

 





In this paper, to address the above-mentioned challenges in unsupervised rPPG estimation, we propose a novel De-interfered and Descriptive rPPG Estimation Network (DD-rPPGNet) to eliminate interference from interference-carrying rPPG features for learning genuine rPPG signals. 
Figure <ref> (b) illustrates our main idea. 
As noted in <cit.>, interference in foreground and background are approximately similar if they are resulted from the same source. In Section  <ref>, we will also verify that foreground and background interference signals indeed exhibit local spatial-temporal similarities.
Therefore, we adopt these characteristics of interference between foreground facial region and background non-facial region to develop an unsupervised interference estimation method.  
Next, we propose an unsupervised de-interfered rPPG estimation to learn genuine rPPG signals with two stages. 
In the first stage, we use contrastive learning to estimate the initial rPPG signals from both the training data and their weakly augmented counterparts. 
In the second stage, we use the estimated interference features to derive de-interfered rPPG features and also encourage the refined rPPG signals to be distinct from the interference.
In addition, to tackle the challenge caused by subtle chrominance changes, we further propose an effective descriptive rPPG feature learning by extending our previously developed 2DLDC <cit.> to develop a 3D Learnable Descriptive Convolution (3DLDC) to capture the subtle temporal and chrominance changes for enhancing rPPG estimation. 
We conduct extensive experiments on five public rPPG databases to evaluate the effectiveness of the proposed DD-rPPGNet.
Our experimental results on intra-domain and cross-domain testing demonstrate that the proposed DD-rPPGNet not only outperforms previous unsupervised rPPG estimation methods  but also achieves competitive performance to state-of-the-art supervised rPPG estimation methods.



  
Our contributions are summarized as follows:

∙  
We propose a novel model DD-rPPGNet, focusing on describing interference characteristics and on eliminating interference from rPPG features, to learn genuine rPPG signals for unsupervised rPPG estimation.






∙ 
Based on our analysis of local spatial-temporal interference similarities between foreground and background regions, we propose a novel unsupervised method for interference estimation to substantially improve rPPG estimation.


∙ 
To effectively capture subtle chrominance changes on facial skin, we additionally introduce a robust 3D Learnable Descriptive Convolution (3DLDC) by incorporating learnable local descriptors into 3DCNN for learning descriptive rPPG features. 

 
∙
Extensive experiments demonstrate that DD-rPPGNet outperforms previous unsupervised rPPG estimation methods and achieves competitive performance with state-of-the-art supervised rPPG estimation methods.

 
 
 



§ RELATED WORK 
  



 §.§ Supervised rPPG Estimation
 
Supervised rPPG estimation focuses on learning the rPPG estimation model from paired facial videos and ground truth rPPG signals. 
Earlier approaches have developed various end-to-end architectures to directly learn rPPG relevant features.
For example, in <cit.>, the authors proposed the first end-to-end system to capture motion information between consecutive frames for robust HR measurement under significant moving scenarios. 
Some other end-to-end methods designed different networks, such as deep spatio-temporal network <cit.>, siamese-rPPG network <cit.>, and recent  PhysFormer <cit.>  to improve the accuracy of rPPG estimation.
In <cit.>, the authors proposed to generate augmented training data through extracting and swapping rPPG signals across different videos. 


 


Since remote estimation of physiological signals on skins is highly sensitive to various interference, such as head motions <cit.>, facial expressions <cit.>, video compression artifacts <cit.>, and lighting variations <cit.>,  
many methods <cit.> relied on ground truth rPPG signals to model interference for learning de-interfered rPPG signals.
In particular, in <cit.>, the authors proposed to adopt generative adversarial learning to model the noise distribution of interference.
Also, in <cit.>, the authors proposed to learn interference co-existing in facial and non-facial regions to suppress their shared covariates and to amplify the rPPG signal information.
In addition, in <cit.>, the authors proposed to generate the motion distortion for filtering out the motion-induced measurements.


Moreover, to capture subtle chrominance changes on facial skin, many methods <cit.> included pre-defined and fixed local descriptors into vanilla 3DCNN, such as 3D Temporal Difference Convolution (TDC) <cit.> and 3D Central Difference Convolution (3DCDC) <cit.>, to enhance the representation capacity of vanilla 3DCNN for learning descriptive rPPG features.


   


 §.§ Unsupervised rPPG Estimation
 

Unlike supervised method, unsupervised rPPG estimation aims to learn rPPG estimators solely from facial videos without relying on ground truth rPPG signals.
In <cit.>, building upon on the observation that rPPG signals exhibit slight variations over small time intervals and that human heart rates generally fall between 40 and 250 beats per minute (bpm), the authors proposed 
an unsupervised rPPG estimation via contrastive learning between original videos and their resampled counterparts.
Next, in <cit.>, by further including the observations that rPPG signals exhibit small variations across different spatial regions of the same video and exhibit larger variations across different videos, the authors extended the findings in <cit.> and proposed adopting spatio-temporal constraints to develop an unsupervised method.
Furthermore, in <cit.>, the authors proposed learning the rPPG signals from augmented videos with dissimilar signal frequencies.
In <cit.>, the authors proposed the first non-contrastive learning framework to learn concentrated energy within the heart rate range for unsupervised rPPG estimation.
While these unsupervised rPPG estimation methods <cit.> have shown promising potential, inclusion of de-interfering and descriptive feature featuring remains unexplored.

 



   



§ BACKGROUND AND MOTIVATION
 
  
  


 §.§  Interference Model for rPPG Estimation 
  


In <cit.>, the authors explicitly modeled the observed intensities in foreground facial component x^fg(p) and in background (i.e., non-facial component) x^bg(p) of each pixel p by,
  

    x^fg(p)    =    base(p) + n^fg  +  r,
    
    
    x^bg(p)    =    base(p) + n^bg,


 


where base(p) denotes the base intensity of the pixel p, 
r denotes the rPPG signal, and
n^fg and n^bg denote the interference in the facial and non-facial regions, respectively.
Note that, as mentioned in <cit.>, the interference n^fg and n^bg in Equations (<ref>) and (<ref>) do not cover random noises (e.g., camera sensor noises), because these noises are randomly and independently distributed and thus are not included in the model.

From Equations (<ref>) and (<ref>), if the interference signals n^fg and n^bg are resulted from the same source, e.g., video compression or illumination variations from a flickering light bulb, then we can approximate the foreground interference by the the background interference, i.e., n^fg≈n^bg. 
Therefore, in <cit.>, the authors proposed to learn the interference n^bg from non-facial regions x^bg and then use n^bg to cancel the interference in x^fg for enhancing learning the rPPG signals r.
Similarly, other supervised methods either utilized generative adversarial learning for generating interference <cit.> or adopted motion distortion model <cit.>  
to eliminate the interference for enhancing the rPPG estimation.

Unlike the supervised methods <cit.>, which rely on the ground truth rPPG signals to learn the de-interfered rPPG signals, existing unsupervised methods <cit.> 
are mostly built upon contrastive learning based on numerous assumptions, such as rPGG spatial-temporal similarity and HR range constraint, to directly estimate rPPG signals. 
Since these unsupervised methods <cit.> are
oblivious to the rPPG interference model, their estimated rPPG signals r̂ are highly susceptible to the interference n^fg, i.e., r̂≅r + n^fg. 
In Section <ref>, we will justify this claim by showing that 
the rPPG signals estimated from existing unsupervised rPPG estimation methods are indeed positively and significantly correlated with the interference n^bg in the non-facial background region. In Sections <ref> and <ref>, we will then discuss the characteristics of n^fg and n^fg and those of rPPG signals r, respectively, and will then develop a novel method in Section <ref> to improve estimating the genuine and de-interfered rPPG signals r under unsupervised learning scenario.





 §.§ rPPG Estimation Involving Interference



As mentioned in <cit.>, since foreground and background interference signals usually come from the same source, we can readily use n^bg to approximate n^fg by n^fg≈n^bg.
Here, we will first analyze the correlation between the estimated rPPG signals r̂ and the interference signals n^bg and then will investigate whether the rPPG signals r̂ are corrupted by interference or not.


First, we adopt the pre-trained model <cit.> as the rPPG estimator to extract both the rPPG signals r̂ and the background interference n^bg
from foreground facial region and background non-facial region, respectively.
Next, we measure the running correlation c_r̂, n between r̂ and n^bg by,






  

    c_r̂, n[τ]    =    NC (r̂[v-τ],  n^bg[v]), 
       =    NC (r̂[v̅],  n^bg[v]),

 
  
where NC(a[v],b[v]) = ∑_l^L a[l] · b[l]/√(∑_l^L (a[l])^2  )√(∑_l^L (b[l])^2  ) 
is the normalized correlation between the two signals a[v] and b[v] of length L, r̂[v̅] = r̂[v-τ] is the shifted signal of r̂[v] by τ samples, and τ∈{-(L-1),⋯,0,⋯,L-1} indicates the running lag.







Assuming the estimated rPPG signals r̂ are corrupted by the interference n^fg, i.e., r̂≅r + n^fg,  then we rewrite Equation (<ref>) by substituting r̂ with r + n^fg and have:  

  

    c_r̂,n[τ]  
        ≅ NC ( r[v-τ] +  n^fg[v-τ] ,  n^bg[v])   
       ≅ NC ( r[v̅] +  n^fg[v̅] ,  n^bg[v])   
       ≅α  NC (r[v̅],  n^bg[v]) +  β  NC (n^fg[v̅],  n^bg[v]),

 
 


where α = √(∑_l^L (r[l])^2)/√(∑_l^L (r[l]+ n^fg[l])^2),  β = √(∑_l^L (n^fg[l])^2)/√(∑_l^L (r[l]+n^fg[l])^2), 
and r[l] and n^fg[l] are the l-th samples of r[v̅] and n^fg[v̅], respectively.
Please see the supplementary material for detailed derivations of Equation (<ref>).




In Equation (<ref>), the correlation between r̂ and n^bg can be decomposed into  two terms. 
The first one is the correlation between the genuine rPPG signal r and the estimated interference n^bg; and the second one is the correlation between the two interference signals n^fg and n^bg.
Therefore, if an estimated rPPG signal r̂ involves no or little interference, i.e., r̂≅r, then r̂ should be independent of the interference n^bg (the first term in Equation (<ref>)) and should have little correlation between  n^fg and n^bg (the second term in Equation (<ref>)).
In Figure <ref>, we show the running correlation   c_r̂, n[τ] between r̂ and n^bg and also the running correlation c_r, n[τ] between the ground truth rPPG signal r and n^bg measured by Equation (<ref>). 
From Figure <ref>, we see that r and n^bg are indeed negatively correlated because of their independent nature. 
However, the positive correlations c_r̂, n[τ] between r̂ and n^bg show that the estimated rPPG signals r̂ are highly corrupted by the interference n^fg.





















 §.§ Characteristics of Interference Signals


 



  §.§.§ Local Spatial-Temporal Similarities of Interference
 


As observed in <cit.>, each facial video often has interference originating from the same source, such as compression artifacts <cit.> or lighting variations <cit.>.
Because these sources of interference are relatively stable across the temporal domain, it is reasonable to assume that the interference exhibits temporal similarity within a single video.
On the other hand, when videos are captured under natural light conditions with uneven illuminations, the resultant interference on different spatial regions (e.g., well-illuminated and dimly illuminated regions) usually exhibits varying characteristics.
Since the interference signals in different spatial regions originate from the same source, they should still exhibit local spatial similarities.
Therefore, from the above two observations, we infer that the interference signals in rPPG exhibit local spatial-temporal similarities and will experimentally verify these characteristics below.



  §.§.§ Experimental Verification



To evaluate the local spatial and temporal characteristics of interference, we adopt the off-the-shelf rPPG estimator <cit.> to extract signals from different non-facial regions. Note that, because we extract the signals from non-facial regions x^bg, these signals capture no rPPG information but reflect only the interference n^bg. In Figures <ref> and <ref>, we extract interference signals n^bg_1 and n^bg_2 of non-facial regions bg_1 and bg_2 in a sample video from COHFACE dataset <cit.> and generate the running correlation c_bg_1,bg_2[τ] to analyze the relationship between n^bg_1 and n^bg_2 during the time lag τ.



In Figure <ref>, we first show that the interference signals n^bg_1 and n^bg_2, extracted from two non-overlapped clips of the same non-facial region, display similar waveform and have a strong positive correlation across temporal domain. 
Next, in Figure <ref> (a)-(c), we measure the running correlation between the signals n^bg_1 and n^bg_2 extracted from two well-illuminated regions, two dimly-illuminated regions, and two differently illuminated regions, respectively. The results in Figure <ref> (a) and (b) show that the extracted interference signals are also highly similar and have a high positive correlation. 
By contrast, in Figure <ref> (c), the two interference signals, extracted from the same video under different lighting conditions, exhibit a significantly negative correlation. 
Consequently, the results in Figures <ref> and <ref> experimentally demonstrate that the interference signals in rPPG model
exhibit local spatial-temporal similarity.

   

  



 §.§ Characteristics of rPPG Signals





  §.§.§ Observations of rPPG Signals in Previous Methods



Previous unsupervised rPPG estimation methods <cit.> have explored and validated several observations about characteristics of rPPG, including HR range constraint, spatial-temporal similarities, and dissimilarity across different videos. 


First, as noted in <cit.>, the majority of human heart rates typically fall within the range of 40 to 250 beats per minute (bpm). Accordingly, the authors in <cit.> recommended utilizing this HR range as a constraint for determining rPPG signals with power spectrum densities (PSDs) falling between 0.66 Hz and 4.16 Hz.



Next, in <cit.>, the authors also observed that HR does not change rapidly in a short time and that HR detected from different facial regions should be consistent. This observation implies that rPPG signals exhibit spatial-temporal similarity within a single facial video.



Moreover, in <cit.>, the authors observed that rPPG signals from different subjects have distinct waveforms and PSDs.
Therefore, they adopted this cross-video dissimilarity within the framework of contrastive learning to improve rPPG estimation.





  §.§.§ Consistency in rPPG with Augmentation



Since data augmentation has been popularly adopted to improve model training, we will also incorporate data augmentation in the proposed method. 
In particular, we expect that a facial video and its weakly augmented counterpart through, e.g., rotation, random crop, and flip, should exhibit identical rPPG signals. 
Figure <ref> shows an example that the extracted rPPG signals from a facial video and its rotated version indeed have similar waveforms and PSDs.
Therefore, in addition to the rPPG characteristics mentioned in the previous subsection, we will also enforce the consistency in augmentation to constrain the proposed model.

 
 
  
  


 




§ PROPOSED METHOD



  




In this paper, we propose a novel unsupervised rPPG estimation model, called De-interfered and Descriptive rPPG Estimation Network (DD-rPPGNet), to eliminate interference in rPPG estimation without referring to ground truth rPPG signals.
In Section <ref>, we first give an overview of the proposed DD-rPPGNet. 
Then, in Section <ref>, we present how we adopt the local spatial-temporal characteristics to model the interference signals.
Next, in Section <ref>, we adopt the rPPG characteristics and also refer to the estimated interference to derive the de-interfered rPPG signals.
Finally, in Section <ref>, we develop a 3D Learnable Descriptive Convolution (3DLDC) to enhance learning descriptive rPPG features for further improving rPPG estimation.


  




 


 §.§ Overview of DD-rPPGNet




Figure <ref> shows the proposed DD-rPPGNet, which consists of two branches: one Interference Estimation Branch and one De-interfered rPPG Estimation Branch. 
First, in the Interference Estimation Branch, we adopt the local spatial-temporal similarity of interference mentioned in Section <ref> to learn the interference features f_n and to estimate the interference signals n. 
Next, in the De-interfered rPPG Estimation Branch,
 we adopt the characteristics of rPPG signals mentioned in Section <ref> to learn the inference-carrying rPPG features f̂_r and signals r̂, and then use f_n, f̂_r, and n to learn the de-interfered rPPG signals r.
In addition, we propose an effective 3DLDC by involving learnable descriptors into 3DCNN to capture subtle chrominance changes for learning descriptive rPPG features.
  



 §.§ Unsupervised Interference Estimation

 


In the Interference Estimation Branch, we adopt the local spatial-temporal similarity of interference mentioned in Section <ref> to model the interference signals n for learning the interference features f_n.



First, we describe how we determine the foreground facial component x^fg and the background non-facial component x^bg from an input video x.
As shown in Figure <ref>, we adopt 3DDFA <cit.> to locate the facial region x^fg and then follow the setting described in <cit.> to crop both the left and right regions of x^fg to generate the non-facial background video x^bg with the same size as x^fg. Next, we randomly sample L clips, each having Δ_t frames with a fixed size of h × w, from x^fg and x^bg to generate the input foreground clips C^fg = {c_l^fg∈ℝ^h × w ×Δ_t}_l=1^L and the background clips C^bg = {c_l^fg∈ℝ^h × w ×Δ_t}_l=1^L, respectively.



As shown in Figure <ref>, given the input clips c^fg∈ C^fg and c^bg∈ C^bg, our goal in Interference Estimation Branch is to estimate the interference signal n=E_n ( F_n (c)) via learning the interference feature extractor F_n and the interference estimator E_n .
Assuming n ≈ n^fg≈ n^bg, we propose to estimate n
by constraining the local spatial-temporal similarity between n^fg and n^bg.
Let N^fg={n^fg} and N^bg={n^bg} denote the sets of estimated interference from the input clips C^fg={c^fg} and C^bg={c^bg} within a batch, respectively.
We first define the interference correlation loss ℒ_nc to enforce the local spatial-temporal similarity between n^fg and n^bg by,


    ℒ_nc = 𝔼{ NP( n_fg , n_bg) },



where 𝔼{·} denotes the expectation,
 NP(  n_fg , n_bg ) = 1- Cov(  n_fg , n_bg )/√(Cov(  n_fg ,  n_fg ))√(Cov( n_bg , n_bg )) is the negative Pearson correlation  <cit.> between n_fg and n_bg, 
 and Cov(n_fg , n_bg) denotes  the covariance between n_fg and n_bg. 
 

In addition, to further enforce the local spatial-temporal similarity of interference, we adopt K-Means(·) <cit.> to aggregate the estimated interference signals into K clusters N_k, k=1,…,K. Next, we define the K-Means-based contrastive interference loss ℒ_kcn to pull together similar signals from the same cluster and to push away relatively  dissimilar signals from other clusters by,

 

    ℒ_kcn = 𝔼{log (exp(NP(n_i,n_j) /∑_n_h ∈  N^fg∪ N^bgexp(NP(n_i,n_h) ) + 1) } ,


 
where n_i, n_j ∈ N_k are the estimated interference signals from the same clusters, and
n_h ∈  N^fg∪ N^bg denotes all the estimated interference signals.

   


 §.§ Unsupervised De-interfered rPPG Estimation

 
 

In the De-interfered rPPG Estimation Branch, we leverage the characteristics of rPPG signals described in Section <ref> and refer to the interference features f_n extracted in the Interference Estimation Branch to directly estimate the rPPG signals without relying on the ground truth.



First, we conduct weak augmentation, including rotation and flip operations, on the original facial clips C^fg to generate augmented video clips aug-C^fg as an additional input to this branch. From the characteristics of rPPG consistency with augmentation, our goal is to derive the initial estimates of rPPG features f̂_r=F_r(c) and rPPG signals r̂ = E_r(f̂_r) by learning the rPPG feature extractor F_r and the rPPG estimator E_r. 
To achieve this goal, we define the contrastive rPPG loss ℒ_cr to pull together the rPPG signals r̂ extracted from a video clip and its augmented counterpart while simultaneously pushing away the rPPG signals r̂ extracted from different videos by,

 

    ℒ_cr = 𝔼{log( exp(NP(r̂_i,r̂_j) ) /∑_r̂_h ∈R̂_allexp(NP(r̂_i,r̂_h) ) +  1 )  } ,

  



where r̂_i,r̂_j ∈R̂_b denote the estimated rPPG signals from C_b^fg and its augmented counterpart aug-C_b^fg of one single video x^fg_b, respectively; and r̂_h ∈R̂_all  denotes the estimated rPPG signals from all the videos and their augmented counterparts within one batch.







Incorporating with the Interference Estimation Branch, our next goal is to extract de-interfered rPPG features f_r by canceling the interference features f_n from the initial estimates f̂_r so as to derive de-interfered rPPG signal by r = E_r(f_r). 
Our inspiration comes from a recent supervised rPPG method <cit.>, which proposed to learn de-noised rPPG features through a residual method by subtracting noise features from the noise-carrying rPPG features.
To expand upon this concept in our unsupervised learning model, we propose to derive de-interfered rPPG features by f_r = f̂_r - f_n^fg, where f_n^fg = F_n (c^fg) is the interference features extracted from the input facial clips. After obtaining the de-interfered rPPG features f_r, we next derive a new set of rPPG signals R = {r_i } from C^fg∪ aug-C^fg, where r = E_r(f_r) is the refined estimate of rPGG signals.





Subsequently, similar to Equation (<ref>), we once again adopt the idea of contrastive learning by replacing the set of initial estimates R̂ with the set of refined estimates R.

Furthermore, we additionally encourage the refined estimates r to be away from the interference n and define the de-interfered contrastive rPPG loss ℒ_dcr by,
 

    ℒ_dcr = 𝔼{log( exp(N  P(r_i,r_j))/∑_r_h  ∈R_allexp( N P(r_i, r_h)  )  + ∑_n_h  ∈  N_fgexp(   N  P(r_i,  n_h) ) +  1 ) },





where r_i,r_j∈ R_b are the refined rPPG signals estimated from C_b^fg and aug-C_b^fg of the input video x^fg_b, respectively; n_h ∈ N_fg denotes the estimated interference from C_b^fg, and 
r_h ∈ R_all  denotes the refined rPPG signals estimated from all the videos and their augmented counterparts within one batch.













 §.§  Descriptive rPPG Feature Learning


  

To extend the description capability of CNN and Transformer, most previous rPPG estimation methods <cit.> adopted predefined descriptors, such as 3D Central Difference Convolution (3DCDC) <cit.> in CNN, to capture 
 the temporal differences of gradient information.

Because these predefined descriptors <cit.> cannot well adapt to subtle changes along temporal domain on uneven skin caused by varying reflection changes <cit.>, we believe including a learnable descriptor into 3D convolution
would positively enhance the rPPG feature descriptive ability.
To enable the model to adaptively focus on blood volume changes in optical information, we extend our previously proposed 2DLDC in <cit.> and propose a novel 3D Learnable Descriptive Convolution (3DLDC) to learn descriptive rPPG features. 










  §.§.§ 2D Learnable Descriptive Convolution and 3D Temporal Difference Convolution

 


We first recap our previous design of 2DLDC <cit.> and the operation of 3D Temporal Difference Convolution <cit.>. As mentioned in <cit.>, predefined and unlearnable descriptors are inflexible to capture various textural changes across different materials and limit the representation capacity of vanilla 2D convolution.
Therefore, in 2D Learnable Descriptive Convolution (2DLDC) <cit.>, we included a learnable local descriptor into vanilla 2D convolution to adaptively capture subtle textures on different materials and thus increase its representation capacity. 
The design of 2DLDC <cit.> is to incorporate a learnable descriptor m into vanilla convolution by,

  

    g(p^t) =    (1-ϵ)∑_p_k^t ∈ℛ^t  w(p_k^t) · f(p^t + p_k^t)_vanilla    convolution
       + ϵ∑_p_k^t ∈ℛ^t  w(p_k^t) · ( f(p^t + p_k^t) 
    · m(p_k^t))_learnable   descriptive   convolution,

 

where w is the convolution kernel, f is the input feature map, p^t is the pixel of current location in the t-th frame, p_k^t is the location of neighboring pixels in a local neighborhood ℛ^t = {(-1,-1),(-1,0),...,(0,1),(1,1)}, g is the output feature map, and the hyperparameter ϵ∈  [0, 1] 
accounts for tradeoffs between
vanilla and 2D LDC convolution.  
  


Although 3D convolution (3DCNN) has been widely employed in rPPG estimation task, vanilla 3DCNN often leads to poor performance under challenging conditions due to its limited representation capacity.
Therefore, 
the authors in <cit.>  proposed Temporal Difference Convolution (TDC) by adopting a predefined descriptor to involve the temporal differences information into 3DCNN to provide fine-grained temporal context for robust rPPG estimation. 

As shown in Figure <ref> (a), TDC aggregates the  temporal difference clues within local temporal regions  ℛ^t-1, ℛ^t, and ℛ^t+1 on the feature level by,

    g(p^t) =   ∑_p^t-1_k ∈ℛ^t-1 
    w(p_k^t-1) · (f(p^t-1+ p_k^t-1) -θ· f(p^t)) 
       +∑_p^t+1_k ∈ℛ^t+1 
    w(p_k^t+1) · (f(p^t+1+ p_k^t+1) -θ· f(p^t)) 
       +∑_p^t_k ∈ℛ^t 
    w(p_k^t) · f(p^t + p_k^t),
 

where the hyperparameter θ∈  [0, 1] specifies the contribution of temporal difference. 





 






  §.§.§ 3D Learnable Descriptive Convolution




To effectively learn descriptive rPPG features, as shown in Figure <ref> (b), we extend our previously proposed 2DLDC <cit.>  and  propose an effective 3D Learnable Descriptive Convolution (3DLDC)   by  incorporating a learnable descriptor m into vanilla 3D convolution to capture subtle skin color changes by,


    g(p^t)
    =     (1 -ϵ) ∑_i∑_p^t+ i_k∈ℛ^t+ i w(p_k^t+ i) · f(p^t+ i+ p_k^t+ i)_vanilla    convolution
       +ϵ∑_i∑_p^t + i_k∈ℛ^t+ i w(p_k^t+ i) · (f(p^t+ i+ p_k^t+ i) · m(p_k^t+ i))_3D   learnable   descriptive   convolution,
  


where the learnable descriptor m and the convolution kernel w are both of the same size 3 × 3 × 3 and i ∈{-1,0,1}.

Note that, the proposed 3DLDC exhibits a good generalization of other convolutions. 
When ϵ=0, 3DLDC apparently becomes vanilla 3D convolution. 
In addition,  by comparing Equation  (<ref>) with Equation  (<ref>), we show that Temporal Difference Convolution (TDC) <cit.> is a special case of 3DLDC when  the matrix m in Equation (<ref>) is  

 

    m = 1_3 × 3 × 3 + 
      [  
          m_0,
          [ [   0   0   0;   0 w_s   0;   0   0   0 ]],
           m_0
    ],




where


    w_s = -1/w(p^t) (
    ∑_p^t-1_k∈ℛ^t-1 w(p^t-1_k)+∑_p^t+1_k∈ℛ^t+1 w(p^t+1_k))





is the weight of the special case in 3DLDC, m_0 is the all-zero matrix 0_3 × 3, and the base matrix 1_3 × 3  × 3 is an all-ones matrix. Detailed derivations of Equation (<ref>) are given in the supplementary material. 
To incorporate 3DLDC into DD-rPPGNet, we then replace all the vanilla 3DCNN with the proposed 3DLDC.



  



§ EXPERIMENTS



 §.§ Experimental Setting



  §.§.§ Datasets and Evaluation Metrics

We  conduct experiments on the following rPPG databases:   UBFC-rPPG <cit.>  (denoted as U), PURE <cit.> (denoted as P), and COHFACE  <cit.> (denoted as C), MR-NIRP <cit.> (denoted as M), and  VIPL-HR <cit.> (denoted as V).

To have a fair comparison with previous methods <cit.>,  we follow the same setting in <cit.> to conduct intra-testing and use the benchmark proposed in <cit.> to conduct cross-testing.
We report all the results in terms of the following evaluation metrics: Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Pearson Correlation Coefficient (R). 

 


  §.§.§ Implementation Details
        
We implement the proposed DD-rPPGNet by Pytorch and empirically set P_n= 4, P_r = 2, D =4 in this paper.  
To train DD-rPPGNet, we set a constant learning rate of 1𝐞-5 with AdamW  <cit.> optimizer for 100 epochs. 

During the inference stage, we follow previous unsupervised methods <cit.> to decompose each test video into non-overlapping 10s clips and use the estimated rPPG signals of each clip to  calculate the power spectrum densities (PSD) for deriving HR of each clip.

 













 



















    


 

 


 §.§ Ablation Study



  §.§.§ 
Comparison between Different Loss Terms and Modules

 
Because C is a challenging dataset involving significant lighting variations on the facial regions, in Table <ref>, we compare using  different combinations of modules and losses on the protocols of intra-testing C→C and cross-testing  U+P→C. 
In addition, in Figure <ref>, we visualize their corresponding gradient-based saliency maps <cit.> to compare the  effectiveness of different rPPG models learned by these combinations.

First, we compare the first case (i.e. interference-carrying rPPG estimation using 3DCNN + ℒ_ĉr̂) with the second case (i.e., the de-interfered rPPG  estimation using 3DCNN + ℒ_ĉr̂ + ℒ_nc+ℒ_kcn+ℒ_dcr).
The quantitative results in the first two rows of Table <ref> shows that the second case indeed improves the estimation of the first one by including the de-interfered feature learning to mitigate the interferences within the estimated rPPG signals.  
Also, as shown in Figures <ref> (b) and (c), we see the saliency maps of the interference-carrying rPPG estimation only focus on partial facial regions due to the challenging illumination variations; whereas the saliency maps of the de-interfered rPPG estimation extensively covers even the dimly illuminated regions.  









Next, we compare the two cases of interference-carrying rPPG estimation using non-descriptive feature learning (i.e., the first case using 3DCNN + ℒ_ĉr̂) with the one using descriptive feature learning (i.e., the third case using 3DLDC + ℒ_ĉr̂).
The results in Table <ref> show that, by including descriptive feature learning into DD-rPPGNet, the third case substantially outperforms the first one.
Figures <ref> (b) and (d) also demonstrate that, because 3DLDC better captures subtle color changes on facial skin under varying reflections, inclusion of descriptive feature learning in Figure <ref> (d) indeed yields more accurate saliency maps concentrated on facial regions. 









Finally, when including descriptive feature learning into de-interfered rPPG estimation in DD-rPPGNet (i.e., the fourth case using 3DLDC + ℒ_ĉr̂ + ℒ_nc+ℒ_kcn+ℒ_dcr), we see that the performance in the fourth row of Table <ref> and the saliency maps in Figure <ref> (e) are greatly improved over the other cases and
 demonstrate the effectiveness of the proposed DD-rPPGNet.












  §.§.§ Comparison between Different Convolutions for Learning Descriptive Features 
  
In Table <ref>, we compare using different 3D convolutions, including TDC <cit.>, 3DCDC <cit.> and the proposed 3DLDC, to replace vanilla 3DCNN in DD-rPPGNet under the same training loss ℒ_ĉr̂ + ℒ_nc+ℒ_kcn+ℒ_dcr
 and then test on the protocols of C→C  and U+P→C. 
The results in Table <ref> show that 3DLDC substantially outperforms the others and demonstrates that the proposed learnable descriptor successfully captures subtle temporal and color changes for learning descriptive rPPG features.








 









































































 


  §.§.§ Effectiveness of De-interfered rPPG Estimation


To evaluate the effectiveness of the proposed de-interfered rPPG estimation, we design five protocols of different interference conditions, including head motion, facial expressions, compression artifacts, lighting variations, and periodic noises, and conduct experimental comparisons in Table <ref>. Detail description of these protocols is given in the supplementary material.
As shown in Table <ref>, in the cases of protocols P1 and P2, the proposed DD-rPPGNet remains resilient to the interference caused by head motion and expression changes and outperforms previous unsupervised rPPG estimation methods <cit.>. Furthermore, in the cases of challenging interference such as compression artifacts, illumination variations, and periodic noises in P3-P5,
we observe that these methods <cit.> have a significant decrease in performance; whereas DD-rPPGNet successfully resists the interference, thanks to the proposed de-interfered rPPG estimation, and largely outperforms these methods.








 




 



















 §.§ Intra-Domain and Cross-Domain Testing

 
In Tables <ref> and <ref>, we conduct intra-domain and cross-domain testing  and compare the results with previous supervised and unsupervised methods. 



  §.§.§ Intra-Domain Testing
 
In Table <ref>, we conduct intra-domain testing on five rPPG datasets, U, P, C, M, and V, where U, P, and M involve minor interference and the other two datasets C and V are more challenging with significant illumination variations. 
First, we see that DD-rPPGNet outperforms all the previous supervised and unsupervised rPPG estimation methods on U, P, and M under minor interference.
Next, when facing challenging datasets C and V, DD-rPPGNet still outperforms all the previous unsupervised methods and achieves comparable performances to the state-of-the-art supervised method.
In particular, as shown in Figure <ref> (e), DD-rPPGNet effectively tackles the challenges posed by lighting variations in C through de-interfered rPPG estimation and descriptive feature learning.
As to the challenging large-scale dataset V, which is collected across nine distinct scenarios and encompasses various variations in face pose, scale, and illumination conditions, previous unsupervised methods have a noticeable performance drop, whereas the proposed DD-rPPGNet remains and continues to yield promising results.










   
 


  §.§.§ Cross-Domain Testing
 
In Table <ref>, we follow the benchmark proposed in <cit.> to conduct cross-domain testing on U, P, and C by using two datasets for training and the remaining one for testing. 
The results in Table <ref> show that DD-rPPGNet yields the best results on the protocols U+C→P and U+P→C, and achieves comparable performance with the state-of-the-art method RErPPG-Net <cit.> on P+C→U.
Note that, RErPPG-Net <cit.>, by using ground truth rPPG signals to augment the training data, indeed performs well when there exists little domain gap between the training and testing domains, such as the protocols P+C→U and U+C→P. However, RErPPG-Net <cit.> performs relatively poorly on the protocol U+P→C because of the significant difference between the training and testing domains.
By contrast, with the de-interfered rPPG estimation and descriptive feature learning, the proposed DD-rPPGNet successfully achieves stable and promising results even on the challenging cross-domain protocol U+P→C. 









  




§ CONCLUSION




In this paper, we propose a novel fully unsupervised rPPG estimation network, called DD-rPPGNet, to tackle the challenges of rPPG estimation in the presence of interference.
First, we investigate the local spatial-temporal characteristics of interference and propose an interference estimation method to model the interference in rPPG signals.
Next, we propose a de-interfered rPPG estimation method by enforcing the rPPG signals to be distinct from the estimated interference.
Furthermore, we propose an effective descriptive rPPG feature learning to capture the subtle color changes on skin to enhance rPPG estimation.  
 Extensive experiments demonstrate that DD-rPPGNet outperforms previous unsupervised rPPG estimation methods and achieves competitive performance with state-of-the-art supervised rPPG estimation method.  





§ APPENDIX



In this appendix, in Section <ref>,  we first present the detailed derivations elucidating the nature of the cross-correlation between rPPG and interference signals.
Next, in Section <ref>, we show that Temporal Difference Convolution (TDC) <cit.> is a special case of the proposed 3D Learnable Descriptive Convolution (3DLDC).
Finally,  in Section <ref>, we provide the detailed description of the proposed protocols.


  




 §.§ Derivation: the nature of the cross-correlation between rPPG and interference signals 
 

 









We perform a running correlation between the interference-carrying rPPG signal r̂ and the estimated interference signal n^bg to calculate their correlation c_r̂, n by, 
  

    c_r̂, n[τ]    =    NC (r̂[v-τ],  n^bg[v]), 
       =    NC (r̂[v̅], n^bg[v]),

  

where 
  

    NC(a[v],b[v]) = ∑_l^L a[l] · b[l]/√(∑_l^L (a[l])^2  )√(∑_l^L (b[l])^2  )



is the operation of normalized correlation between the signals a[v] and  b[v],
L is the length of signal, a[l] is the l-th sample of the signal a[v], c_r̂,n[τ] is the τ-th sample of the correlation signal  c_r̂,n, 
r̂[v̅] = r̂[v-τ] is the signal r̂[v] shifted by τ samples, τ=-(L-1),⋯,0,⋯,L-1 is the lag.
Note that, when τ>0, r̂[v-τ] is equal to the signal r̂[v] shifted u samples to the right,  while when τ<0, r̂[v-τ] is equal to the signal r̂[v] shifted τ samples to the left.
In addition, we use zero-padding to keep the same length between r̂[v̅] and  n^bg[v].






Considering the interference-carrying rPPG signals r̂,  which consist of the genuine rPPG signal r and the interference signals n^fg, we submit r̂ = r + n^fg into (<ref>) to rewrite Equation (<ref>) as,  

  

    c_r̂,n[u]  
        =    NC ( r[v-τ] +  n^fg[v-τ] ,   n^bg[v])  
       =    NC ( r[v̅] +  n^fg[v̅] ,  n^bg[v])  
       =   ∑_l^L (r[l]+n^fg[l]) · (n^bg[l])/√(∑_l^L (r[l] +  n^fg[l])^2)√(∑_l^L (n^bg[l])^2  )
       =   ∑_l^L r[l] ·n^bg[l]/√(∑_l^L (r[l] +  n^fg[l])^2  )√(∑_l^L (n^bg[l])^2  )
           + ∑_l^L n^fg[l] ·n^bg[l]/√(∑_l^L (r[l] +  n^fg[l])^2  )√(∑_l^L (n^bg[l])^2  )
       =   √(∑_l^L (r[l])^2)/√(∑_l^L (r[l] +  n^fg[l])^2  )·∑_l^L r[l] ·n^bg[l]/√(∑_l^L (r[l])^2 )√(∑_l^L (n^bg[l])^2 )
          + √(∑_l^L (n^fg[l])^2)/√(∑_l^L (r[l] +  n^fg[l])^2  )·∑_l^L n^fg[l] ·n^bg[l]/√(∑_l^L (r[l])^2 )√(∑_l^L (n^bg[l])^2 )
       =   √(∑_l^L (r[l])^2)/√(∑_l^L (r[l] +  n^fg[l])^2  )· NC (r[v̅],  n^bg[v]) 
          + √(∑_l^L (n^fg[l])^2)/√(∑_l^L (r[l] +  n^fg[l])^2  )· NC (n^fg[v̅],  n^bg[v]) 
       =   α  NC (r[v̅],  n^bg[v]) +   β NC (n^fg[v̅],  n^bg[v]),


 
where α = √(∑_l^L (r[l])^2)/√(∑_l^L (r[l]+ n^fg[l])^2) and β = √(∑_l^L (n^fg[l])^2)/√(∑_l^L (r[l]+n^fg[l])^2), 
r[l] is the l-th sample of the signal r[v̅], 
 n^fg[l] is the l-th sample of the signal n^fg[v̅].



To evaluate the correctness of Equation (<ref>), we compare the correlation signals produced by  Equation (<ref>) and  Equation (<ref>). 
In particular, we consider using the ground truth rPPG signal as r and adopting the off-the-shelf rPPG estimator <cit.> to extract interference  signals from the two different non-facial regions, denoted as n^fg and n^bg.
Next, we propose to combine r and n^fg to  simulate the interference rPPG signal r̂ = r + n^fg.
Finally, we  Equation (<ref>) and Equation (<ref>) to calculate their correlations, shown in Figure <ref>.
We see that the correlations produced by using  Equation (<ref>) and  Equation (<ref>) yield nearly identical results. 

 


 §.§ Derivation: TDC as a Special Case of 3DLDC

 

We first <cit.> to formulate  Temporal Difference Convolution (TDC) within local temporal regions  ℛ^t-1, ℛ^t, and ℛ^t+1 on the feature level by,



    g(p^t)    = ∑_p^t-1_k ∈ℛ^t-1 
    w(p_k^t-1) · (f(p^t-1 + p_k^t-1) - θ· f(p^t)) 
       +∑_p^t+1_k ∈ℛ^t+1 
    w(p_k^t+1) · (f(p^t+1 + p_k^t+1) - θ· f(p^t)) 
       +∑_p^t_k ∈ℛ^t 
    w(p_k^t) · f(p^t + p_k^t),



where w is the convolution kernel, f is the input feature map, p^t is the pixel of current location in the t-th frame,  p_k^t is the location of neighboring pixels in ℛ^t = {(-1,-1),(-1,0),...,(0,1),(1,1)},  
g is the output feature map, and  θ is a  hyperparemter for controling the contribution of temporal difference.

Next, we propose the 3D Learnable Descriptive Convolution (3DLDC) to adaptively focus on blood volume changes in optical information for learning distinctive rPPG features by, 


    g(p^t)
       =(1 - ϵ) ∑_i∑_p^t+i_k∈ℛ^t+i w(p_k^t+i) · f(p^t+i + p_k^t+i)_vanilla    convolution
       +ϵ∑_i∑_p^t+i_k∈ℛ^t+i w(p_k^t+i) · (f(p^t+i + p_k^t+i) · m(p_k^t+i))_3D learnable   descriptive   convolution



where  m is the learnable descriptor,  w is  the convolution kernel, and ϵ is a hyperparameter for controling the contribution of 3D LDC.
Note that,  m and  w are both of the same size 3 × 3 × 3.

We show that Temporal Difference Convolution (TDC) <cit.> is a special case of 3DLDC when  the matrix m in Equation (<ref>) is 


    m = 1_3 × 3 × 3 + 
      [  
          m_0,
          [ [   0   0   0;   0 w_s   0;   0   0   0 ]],
           m_0
    ],



where


    w_s = -1/w(p^t) (
    ∑_p^t-1_k∈ℛ^t-1 w(p^t-1_k)+∑_p^t+1_k∈ℛ^t+1 w(p^t+1_k))



is the weight for special case of 3DLDC, m_0 is all-zero matrix 0_3 × 3,
and the base matrix 1_3 × 3  × 3 is an all-ones matrix.

 
By merging vanilla convolution and 3DLDC, we rewrite Equation (<ref>) as, 


    g(p^t)    = ∑_i=-1^1 ∑_p^t+i_k ∈ℛ^t+iŵ(p_k^t+i) · f(p^t+i + p^t+i_k),



where 


    ŵ   = [ [ (1 - ϵ)w_-1,-1^t-1 + ϵ w_-1,-1^t-1 m_-1,-1^t-1                                              ⋯                                              ⋮;                                              ⋮                                              ⋱                                              ⋮;                                              ⋮                                              ⋯       (1 - ϵ)w_1,1^t-1 + ϵ w_1,1^t-1 m_1,1^t-1;                                                ],;                                                             [ (1 - ϵ)w_-1,-1^t + ϵ w_-1,-1^t m_-1,-1^t                                        ⋯                                        ⋮;                                        ⋮                                        ⋱                                        ⋮;                                        ⋮                                        ⋯       (1 - ϵ)w_1,1^t + ϵ w_1,1^t m_1,1^t;                                          ],;  [ (1 - ϵ)w_-1,-1^t+1 + ϵ w_-1,-1^t+1 m_-1,-1^t+1                                              ⋯                                              ⋮;                                              ⋮                                              ⋱                                              ⋮;                                              ⋮                                              ⋯       (1 - ϵ)w_1,1^t+1 + ϵ w_1,1^t+1 m_1,1^t+1;                                                ] ]
     
       =[ [ w_-1,-1^t-1(1-ϵ+ϵ m_-1,-1^t-1)                              ⋯                              ⋮;                              ⋮                              ⋱                              ⋮;                              ⋮                              ⋯     w_1,1^t-1(1-ϵ+ϵ m_1,1^t-1);                                ],;                                         [ w_-1,-1^t(1-ϵ+ϵ m_-1,-1^t)                          ⋯                          ⋮;                          ⋮                          ⋱                          ⋮;                          ⋮                          ⋯     w_1,1^t(1-ϵ+ϵ m_1,1^t);                            ],;                                  [ w_-1,-1^t+1(1-ϵ+ϵ m_-1,-1^t+1)                              ⋯                              ⋮;                              ⋮                              ⋱                              ⋮;                              ⋮                              ⋯     w_1,1^t+1(1-ϵ+ϵ m_1,1^t+1) ] ].






where  w^t+i_p_k^ = w(p^t+i_k), p_k^∈{(-1,-1),  ...,   (1,1)}. 
By substituting m in Equation (<ref>) into  Equation (<ref>), we rewrite Equation (<ref>) as, 







    ŵ = [                                         [ w_-1,-1^t-1(1-ϵ+ϵ)                  ⋯                  ⋮;                  ⋮                  ⋱                  ⋮;                  ⋮                  ⋯   w_1,1^t-1(1-ϵ+ϵ);                    ],; [       w_-1,-1^t(1-ϵ+ϵ)                      ⋯                      ⋮;                      ⋮ w_0,0^t(1-ϵ+ϵ (1+w_s))                      ⋮;                      ⋮                      ⋯         w_1,1^t(1-ϵ+ϵ);                        ],;                                          [ w_-1,-1^t+1(1-ϵ+ϵ)                  ⋯                  ⋮;                  ⋮                  ⋱                  ⋮;                  ⋮                  ⋯   w_1,1^t+1(1-ϵ+ϵ);                    ] ]
    
    = [                                                   [ w_-1,-1^t-1           ⋯           ⋮;           ⋮           ⋱           ⋮;           ⋮           ⋯   w_1,1^t-1;             ],; [        w_-1,-1^t                ⋯                ⋮;                ⋮ w_0,0^t(1+ϵ w_s)                ⋮;                ⋮                ⋯          w_1,1^t;                  ],;                                                    [ w_-1,-1^t+1           ⋯           ⋮;           ⋮           ⋱           ⋮;           ⋮           ⋯   w_1,1^t+1;             ] ]












































where





    w_0,0^t(1+ϵ w_s)   = w_0,0^t + ϵ w_0,0^t w_s 
       = w_0,0^t - ϵ (
    ∑_p^t-1_k∈ℛ^t-1 w(p^t-1_k)+∑_p^t+1_k∈ℛ^t+1 w(p^t+1_k))





By  substituting Equation (<ref>) into  Equation (<ref>), we rewrite Equation (<ref>) as,




    g(p^t)    = ∑_i=-1^1 ∑_p^t+i_k ∈ℛ^t+i w(p_k^t+i) · f(p^t+i + p^t+i_k)_vanilla    convolution
       - ϵ· (∑_p^t-1_k ∈ℛ^t-1 w(p_k^t-1) + ∑_p^t+1_k ∈ℛ^t+1 w(p_k^k+1) ) · f(p^t)._TDC
       = ∑_p^t-1_k ∈ℛ^t-1 
    w(p_k^t-1) · (f(p^t-1 + p_k^t-1) - ϵ· f(p^t)) 
       +∑_p^t+1_k ∈ℛ^t+1 
    w(p_k^t+1) · (f(p^t+1 + p_k^t+1) - ϵ· f(p^t)) 
       +∑_p^t_k ∈ℛ^t 
    w(p_k^t) · f(p^t + p_k^t),




When ϵ=θ, 3DLDC apparently becomes TDC in Equation (<ref>). 




 §.§ The proposed protocols
 



In this section, we provide the detailed description of the proposed protocols. 
In protocol 1, we use  the training and testing data involved rapid head movements from  the original setup "Fast Translation" in the dataset PURE  <cit.>.
Next, in protocol 2, we use the training and testing data involved  facial expressions from the original setup "Talking" in PURE.
Moreover, in protocol 3, we the training data from the original setup  "Steady" in PURE.
To simulate compression artifacts, we compress the testing videos from the same  setup "Talking" in  PURE 10 times by adopting a JPEG ratio of 10:1 as the testing data. 
Furthermore, in protocol 4, we use the training and testing data from  the original setup "NATURE" in the dataset COHFACE dataset <cit.>.
Finally, in protocol 5, we use the training data from the  dataset UBFC-rPPG  <cit.>, and follow <cit.> to add the periodic interferences into both the facial region and non-facial regions of testing videos from UBFC-rPPG  as the testing data.



IEEEtran
 



 Paper Title: RoadFormer+: Delivering RGB-X Scene Parsing through Scale-Aware Information Decoupling and Advanced Heterogeneous Feature Fusion.







RoadFormer+: Delivering RGB-X Scene Parsing through Scale-Aware Information Decoupling and Advanced Heterogeneous Feature Fusion
    Jianxin Huang^0009-0004-6558-9589, Student Member, IEEE, Jiahang Li^0009-0005-8379-249X, Graduate Student Member, IEEE, Ning Jia, 
Yuxiang Sun^0000-0002-7704-0559, Chengju Liu^0000-0001-7543-0855, Qijun Chen^0000-0001-5644-1188, Senior Member, IEEE, and Rui Fan^0000-0003-2593-6596, Senior Member, IEEE


^
        

    August 5, 2024
================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT Task-specific data-fusion networks have marked considerable achievements in urban scene parsing. Among these networks, our recently proposed RoadFormer successfully extracts heterogeneous features from RGB images and surface normal maps and fuses these features through attention mechanisms, demonstrating compelling efficacy in RGB-Normal road scene parsing. However, its performance significantly deteriorates when handling other types/sources of data or performing more universal, all-category scene parsing tasks. To overcome these limitations, this study introduces RoadFormer+, an efficient, robust, and adaptable model capable of effectively fusing RGB-X data, where “X”, represents additional types/modalities of data such as depth, thermal, surface normal, and polarization. Specifically, we propose a novel hybrid feature decoupling encoder to extract heterogeneous features and decouple them into global and local components. These decoupled features are then fused through a dual-branch multi-scale heterogeneous feature fusion block, which employs parallel Transformer attentions and convolutional neural network modules to merge multi-scale features across different scales and receptive fields. The fused features are subsequently fed into a decoder to generate the final semantic predictions. Notably, our proposed RoadFormer+ ranks first on the KITTI Road benchmark and achieves state-of-the-art performance in mean intersection over union on the Cityscapes, MFNet, FMB, and ZJU datasets. Moreover, it reduces the number of learnable parameters by 65% compared to RoadFormer. Our source code will be publicly available at <mias.group/RoadFormerPlus>.




urban scene parsing, heterogeneous features, Transformer, convolutional neural network.






§ INTRODUCTION



 §.§ Background

SCENE parsing, also known as semantic segmentation, is crucial for the safety of autonomous driving <cit.>. With the widespread adoption of deep learning techniques, convolutional neural networks (CNNs) and Transformers have demonstrated significant performance improvements over traditional geometry-based models in various image segmentation tasks <cit.>. However, single-modal networks that rely solely on RGB images show limitations in handling challenging conditions such as poor illumination and adverse weather <cit.>. To tackle these problems, subsequent research has explored the integration of useful information provided by additional data modalities. Depth or surface normal information has been utilized to identify spatially continuous regions <cit.>, while thermal images have been employed to enhance object recognition robustness under poor lighting conditions <cit.>. Furthermore, polarization information has been used to improve segmentation performance for transparent and highly reflective objects <cit.>. Our recently proposed RoadFormer <cit.> effectively extracts heterogeneous features from RGB images and surface normal information and fuses these features for robust urban scene parsing, demonstrating notable efficacy in freespace and road defect detection. However, RoadFormer still has several limitations, especially when handling other types/sources of data. Moreover, the large quantity of parameters leads to considerable hardware resource consumption, thus limiting its deployment on terminal devices.



 §.§ Existing Challenges and Motivation

Most existing data-fusion networks use symmetric duplex encoders to extract heterogeneous features from multiple data sources and fuse them to provide a more comprehensive understanding of the environment <cit.>. However, while prior arts <cit.> have been developed to capture more discriminative features using these weight-separating duplex encoders, directly fusing these features may produce ambiguous features, thus negatively impairing the performance of scene parsing <cit.>. Additionally, the symmetric models with extensive parameters require more hardware resources for training, particularly when compared to networks that rely solely on RGB images <cit.>. Therefore, exploring an efficient and effective heterogeneous feature encoding strategy remains an under-explored research area that deserves more attention.

In addition to the heterogeneous feature extraction strategy, the performance of a data-fusion network also depends on the manner in which these features are fused. To address this issue, recent works <cit.> employ learnable feature fusion approaches, which significantly outperform traditional, non-discriminative fusion methods that rely solely on element-wise concatenation or summation <cit.>. For example, RoadFormer <cit.> adopts a Transformer-based approach to effectively capture long-range dependencies within heterogeneous features. On the other hand, RDFNet <cit.> employs CNN-based modules to process multi-scale features, effectively extracting local spatial cues, such as edges and textures, within a relatively small receptive field. However, these methods typically employ single-branch feature fusion blocks, where features extracted from RGB images and additional data types (referred to as “X” data) are fused using convolutional layers or attention mechanisms. Such single-branch feature fusion strategies may not always effectively encode both local and global contexts simultaneously, limiting their capacity to fully exploit the advantages of multi-modal/source data fusion. Considering Transformers' remarkable capability in modeling long-range dependencies and CNNs' robustness in local feature extraction <cit.>, further research into combining CNNs' local features and Transformer's global representations through a dual-branch fusion design to enhance scene parsing is highly warranted.

Moreover, while task-specific networks such as RoadFormer demonstrate impressive performance in RGB-Normal road scene parsing, their applicability to more universal urban scene parsing tasks and their effectiveness in handling diverse data types remain limited. For instance, RoadFormer exhibits a significant performance drop on comprehensive scene parsing datasets, such as the KITTI Semantics <cit.> and Cityscapes <cit.>, compared to existing state-of-the-art (SoTA) RGB-D/Normal methods. Additionally, it performs suboptimally when processing RGB-Thermal/Polarization data <cit.>. It is urged to design a universal RGB-X data-fusion network that performs robustly across multiple data sources for urban scene parsing. 



 §.§ Novel Contributions


To address the aforementioned limitations, we first design a more efficient hybrid feature decoupling encoder (HFDE). Given the correlation between RGB images and their corresponding X data, we first replace the duplex encoder with a weight-sharing backbone to reduce the number of learnable parameters. We then employ an asymmetric architecture that independently utilizes two global feature enhancers (GFEs) and two local feature extractors (LFEs) to decouple heterogeneous features, effectively modeling their inherent differences at various scales.
Subsequently, we introduce a robust dual-branch multi-scale heterogeneous feature fusion (MHFF) block to fuse heterogeneous features in parallel, ensuring a comprehensive integration of global and local features. The MHFF block utilizes Transformer-based and CNN-based modules to parallelly fuse and calibrate multi-scale features. Our proposed RoadFormer+, an upgraded version of RoadFormer, with all these innovative components incorporated, demonstrates superior performance over RoadFormer across four RGB-Normal scene parsing datasets, while reducing the learnable parameters by around 65%. Furthermore, RoadFormer+ achieves SoTA performance in RGB-Normal, RGB-Thermal, and RGB-Polarization scene parsing, demonstrating its exceptional applicability across a broad range of RGB-X data-fusion scenarios.

Our contributions can be summarized as follows:




  * We introduce HFDE, which consists of a weight-sharing backbone and two pairs of independent GFEs and LFEs, to extract heterogeneous features and effectively capture both the correlation and inherent differences between RGB images and X data.


  * We design a dual-branch MHFF block to capture both global and local features simultaneously. It seamlessly integrates Transformer-based and CNN-based modules, so as to utilize different receptive fields to achieve advanced heterogeneous feature fusion.


  * We propose RoadFormer+, a novel urban scene parsing approach with fewer parameters compared with RoadFormer, which achieves SoTA performance across multiple RGB-X scene parsing datasets.





 §.§ Article Structure

The remainder of this article is organized as follows: In Sect. <ref>, we review related works on urban scene parsing. In Sect. <ref>, we introduce our proposed RoadFormer+. In Sect. <ref>, we present quantitative and qualitative experimental results and their corresponding analyses. Finally, in Sect. <ref>, we conclude this work and discuss potential future directions.



§ RELATED WORK





 §.§ Single-Modal Scene Parsing


Since the introduction of fully convolutional network (FCN) <cit.>, various CNN-based scene parsing networks have been developed. For instance, pyramid scene parsing network (PSPNet) <cit.> uses pyramid pooling to capture semantic information at multiple scales. DeepLabV3+ <cit.> employs atrous convolutions with different dilation rates to enrich the contextual feature encoding across scales. Additionally, MobileNetV2 <cit.> adopts lighter architectures based on depth-wise separable convolutions to reduce model parameters to computational demands. In these CNN-based networks, each convolutional kernel processes only a local region of the image at a time. This local receptive field design enables CNNs to excel at extracting local features, such as edges and textures <cit.>.

Transformers have gained prominence in scene parsing tasks due to their exceptional global aggregation capabilities compared to CNNs <cit.>. The attention mechanisms within Transformers allow each token to interact with all others simultaneously <cit.>. These interactions help achieve a comprehensive understanding of the correlation between each token and the global context, thereby better extracting global features. Segmentation Transformer (SETR) <cit.>, pioneering the use of a Transformer-based architecture for scene parsing, adopts a method similar to the vision Transformer (ViT) <cit.> by tokenizing images into patches and processing them through Transformer blocks to enhance the global context modeling in the encoder. Furthermore, the MaskFormer series <cit.> introduces a novel Transformer-based decoding paradigm by segmenting images into a set of masks, each associated with a class prediction. This mask classification paradigm, previously validated in <cit.>, has been effectively incorporated into our enhanced RoadFormer+ design, further optimizing its performance.



 §.§ Data-Fusion Scene Parsing


Scene parsing networks that rely solely on RGB images have been found to be highly sensitive to environmental factors such as lighting and weather conditions <cit.>. To overcome this limitation, data-fusion networks effectively utilize heterogeneous features extracted from RGB images and additional data sources. FuseNet <cit.> pioneered the incorporation of depth information into scene parsing. It uses independent CNN encoders for RGB and depth images and fuses their features through element-wise summation. MFNet <cit.> and RTFNet <cit.> strike a balance between speed and accuracy in RGB-Thermal driving scene parsing. Inspired by <cit.>, the SNE-RoadSeg series <cit.> incorporates surface normal information into freespace detection. These networks employ densely connected skip connections to enhance feature decoding. Despite the improved performance achieved by these networks, the simplistic feature fusion strategies potentially restrict their capacity to fully exploit the complementary information present in heterogeneous features.

To address this challenge, recent studies have employed more advanced and learnable feature fusion strategies. RoadFormer <cit.> combines self-attention with channel attention to form a novel feature synergy block that greatly enhances the fusion of heterogeneous features. Data-fusion networks have also garnered attention in the broader domain of scene parsing. Recent works CMX <cit.> and CAINet <cit.> utilize various attention modules to effectively fuse and recalibrate heterogeneous features. Additionally, SASEM <cit.> introduces a plug-and-play module to enhance semantic supervision, thereby improving feature recovery capabilities in decoding. Moreover, CDDFuse <cit.> implements a two-step training strategy that integrates CNN and Transformer blocks in parallel to fuse multi-modal medical images effectively. This article delves into more robust and general-purpose modules so as to more effectively fuse heterogeneous features. Our proposed RoadFormer+ not only broadens its applicability and generalizability to a wider range of scene parsing tasks but also significantly reduces the number of model parameters.






§ METHODOLOGY


As shown in Fig. <ref>, our proposed RoadFormer+ is composed of three key components:

    
  * An HFDE, which utilizes a weight-sharing backbone to extract heterogeneous features from RGB-X data and decouple them into global and local features using GFEs and LFEs;
    
  * An MHFF block, which employs Transformer-based and CNN-based modules in parallel to recalibrate and enhance global and local features, respectively, and integrates them to generate fused features;
    
  * A Transformer-based decoder that utilizes the fused features to generate final semantic predictions. Given the focus of this study on enhancing feature encoding and fusion strategies, we retain the decoder from the original RoadFormer <cit.>.







 §.§ Hybrid Feature Decoupling Encoder




  §.§.§ Overall Feature Encoding Pipeline


Current networks generally employ symmetric duplex encoders to extract heterogeneous features from multiple data sources <cit.>. However, such dual-branch designs not only double the number of learnable parameters in the feature encoding phase but may also potentially lead to feature conflict <cit.>. To address this issue, we propose an HFDE to improve the efficiency of heterogeneous feature extraction. Specifically, considering the correlation between heterogeneous features <cit.>, we first employ a weight-sharing backbone to process the given RGB image I^R∈ℝ^H× W × 3 and its corresponding X data I^X∈ℝ^H× W × 3, thereby generating multi-scale features ℱ^R={F^R_1, …,F^R_4 } and ℱ^X = {F^X_1, …, F^X_4 }. For X data with a single channel (such as depth, thermal, and polarization information), we replicate it three times along the channel dimension to match the RGB image's dimensions of H× W × 3. Here, F^R, X_i ∈ℝ^H/S_i×W/S_i× C_i represents the features in the i-th encoding stage, where C_i and S_i=2^i+1 (i∈[1,4]∩ℤ) denote the channel and stride numbers, respectively, and H and W denote the height and width of the input image, respectively. Furthermore, we employ two weight-separating GFEs and LFEs to extract global features ℱ_G^R,X and local features ℱ_L^R,X at four spatial scales from the heterogeneous features ℱ^R,X, respectively, thereby realizing feature decoupling. Finally, ℱ_G^R,X and ℱ_L^R,X are fed into the MHFF block for further feature fusion and recalibration.




  §.§.§ Weight-Sharing Backbone
 
Large-kernel convolutions exhibit considerable potential in capturing long-range dependencies, owing to their expansive receptive fields, while still retaining favorable inductive biases crucial for vision-specific tasks such as scene parsing <cit.>. For instance, the areas surrounding vehicles are more likely to be roads rather than buildings. In our previous study <cit.>, ConvNeXt <cit.> demonstrates superior performance compared to ResNet <cit.> and Swin Transformer <cit.>, and thus, we continue to adopt it as the backbone in this study. We also compare the performance of ConvNeXt with the recently proposed SoTA backbone networks UniRepLKNet <cit.> and DiNAT <cit.>. Detailed experimental results and analyses are provided in Table <ref>. Our backbone is constructed using two identical, weight-sharing ConvNeXt models.




  §.§.§ Global Feature Enhancer
 
ViT has shown exceptional performance across various fundamental vision tasks <cit.>. Its self-attention mechanism effectively models the global receptive field, thereby enhancing the contextual understanding essential for recognizing large continuous areas such as roads and sidewalks. Consequently, we utilize a GFE based on the multi-head self-attention mechanism to further emphasize the long-range global features. Given the robustness of the backbone network, we omit the positional encoding and replace the commonly used feed-forward network layers with simple normalization operations to reduce the number of model parameters. RGB features ℱ^R and X features ℱ^X are respectively mapped to query, value, and key matrices through convolutional layers. We also introduce a residual connection into the attention operation. Our GFE module can be formulated as follows:

    G_i  = Norm( MHSA
    ( F_i )  + F_i ),

where F_i represents the i-th feature maps within ℱ^R and ℱ^X, G_i represents the i-th feature maps within ℱ^R_G and ℱ^X_G, and MHSA represents the multi-head self-attention mechanism operation. After processing by the GFE, the enhanced global features ℱ^R_G and ℱ^X_G are obtained.




  §.§.§ Local Feature Extractor
 
Local detail features, such as edges and corners, are crucial for accurate scene parsing. Compared to Transformers, convolution operations are proficient at extracting local features and further enhancing fine-grained details <cit.>. Therefore, we propose a LFE, which incorporates an inverted residual block from MobileNetV2 <cit.> to process ℱ^R and ℱ^X, specifically targeting local features. This lightweight module strikes a balance between model parameters and accuracy, as demonstrated across multiple tasks <cit.>. Our LFE can be formalized as follows:

    L_i
        = 1× 1Conv( ReLU( 3× 3DWConv( ReLU
        (
        1× 1Conv
        (F_i)
        )
        )
        ) 
        ) + F_i,

where L_i denotes the i-th feature maps within ℱ^R_L and ℱ^X_L. After processing by the LFE, we obtain the local features ℱ^R_L and ℱ^X_L.



 §.§ Multi-Scale Heterogeneous Feature Fusion Block


To further fuse and integrate global and local features, we introduce a dual-branch MHFF block, which employs attention mechanisms and CNN modules in parallel. An MHFF consists of (1) a global feature recalibration module (GFRM) that utilizes a cross-attention mechanism to recalibrate ℱ^R_G and ℱ^X_G, (2) a local feature fusion module (LFFM) that utilizes convolutional layers to fuse ℱ^R_L and ℱ^X_L, and (3) a feature enhancement and integration module (FEIM) based on a spatial attention mechanism to integrate heterogeneous features, and generate robust fused feature ℱ^F.






  §.§.§ Global Feature Recalibration Module

Heterogeneous global features ℱ^R_G and ℱ^X_G are generally complementary <cit.>. For example, road areas often appear consistent in color across RGB images and possess uniform normal vectors and polarization properties. Therefore, one feature type can be utilized to mitigate potential noise in its complementary feature type <cit.>. Additionally, not all multi-channel features contribute positively to semantic predictions <cit.>, necessitating the recalibration of heterogeneous features along the channel dimension <cit.>. To address these challenges, we introduce the GFRM (see Fig. <ref> (a)) to recalibrate and fuse ℱ^R_G and ℱ^X_G. The cross-attention mechanism, which considers interactions among all positions in the input, is well-suited for calibrating complementary heterogeneous global features and has demonstrated excellent performance in many visual tasks <cit.>. Drawing inspiration from these approaches, the GFRM first recalibrates global features using a cross-attention mechanism, which can be formulated as follows:

    G^R'_i  = Softmax( Q^R_i K^X_i^⊤) κ_i V^R_i + G^R_i,


    G^X'_i  = Softmax( Q^X_i K^R_i^⊤) γ_i V^X_i + G^X_i,


    F^C_i = Norm( δ(  [G^R'_i , G^X'_i ]  ) ),

where G^R_i and G^X_i represent the i-th feature maps within ℱ^R_G and ℱ^X_G, respectively, G^R_i and G^X_i are then identically mapped to query Q^R,X_i, key K^R,X_i and value V^R,X_i embeddings, [·,·] denotes the concatenation operation along the channel dimension, and δ is a non-linear activation function. The learnable coefficients κ_i and γ_i can adaptively adjust attention significance <cit.>. For ℱ^C, we further employ channel-wise attention to emphasize key features and suppress those with low information density, which can be formulated as follows:

    z_i,j= S_i^2/H W∑_h=1^H/S_i∑_w=1^W/S_iF^C_i(h,w,j),


    F^G_i 
        = F^C_i 
        ⊙σ(
        1× 1Conv
        (z_i)
        )
        + F^C_i,

where z_i=[z_i,1,…,z_i,C_i]∈ℝ^1× 1× C_i stores the average pooling results of each feature map in F^C_i, σ is the sigmoid function, and ⊙ denotes the Hadamard product operation. Finally, we obtain the fused global feature ℱ^G={F^G_1, …,F^G_4 }. Here, F^G_i ∈ℝ^H/S_i×W/S_i× C_i represents the global features in the i-th feature fusion stages.



  §.§.§ Local Feature Fusion Module

To preserve more local contexts when fusing heterogeneous features ℱ^R_L and ℱ^X_L, we propose a convolution-based LFFM (see Fig. <ref> (b)). Inspired by the MLP-Mixer <cit.>, our LFFM captures relationships between heterogeneous features from different local regions, generating fused local features. The LFFM can be mathematically represented as follows:

    H^L_i =  3× 3DWConv( 
        Conv([L^R_i , L^X_i])
        ),

where L^R_i and L^X_i represent the i-th feature maps within ℱ^R_L and ℱ^X_L, respectively, each having C_i channels. After concatenating L^R_i and L^X_i along the channel dimension, we employ depth-wise separable convolutions to expand their channels to 4C_i, thereby enhancing the local context. The resultant H^L_i is then split into H^M_i and H^N_i along the channel dimension. This design allows the model to learn new feature representations, the effectiveness of which is further validated in Table <ref>. These two components interact through Hadamard multiplication, enabling the capture of relationships between features from different local regions:

    F^L_i =  1× 1Conv(
        H^M_i ⊙σ(H^N_i)
        ),

where we utilize the Gaussian error linear unit (GELU) as the activation function σ(·). Then we obtain the fused local features ℱ^L={F^L_1, …,F^L_4 }, where F^L_i ∈ℝ^H/S_i×W/S_i× C_i represents the local features in the i-th feature fusion stage.




  §.§.§ Feature Enhancement and Integration Module

Spatial information is crucial for capturing spatial structures in visual perception tasks <cit.>. Nonetheless, our GFRM and LFFM fuse heterogeneous features across channel dimensions, squeezing spatial information into a channel descriptor, and hence is difficult to preserve positional information <cit.>. Therefore, it is necessary to introduce additional spatial information when integrating global features ℱ^G and local features ℱ^L. The spatial attention mechanism emphasizes the importance of specific regions within features, aiding the network in focusing on “where” informative parts are located <cit.>. Inspired by the coordinated attention <cit.>, we introduce the FEIM (see Fig. <ref> (c)) to further enhance and integrate ℱ^G and ℱ^L, enabling the module to detect more subtle spatial variations. Specifically, we employ global pooling kernels (H, 1) or (1, W) to aggregate features along the height and width dimensions, respectively. Thus, the output of the j-th channel at height p and width q can be formulated as:

    F^S_i = F^G_i + F^L_i,


    z_i,j,p^h = 1/W∑_0 ≤ m < WF^S_i(p, m, j),


    z_i,j,q^w = 1/H∑_0 ≤ n < HF^S_i(n, q, j),

where F^G_i and F^L_i represent the i-th feature maps within ℱ^G and ℱ^L, respectively, and Z^h_i∈ℝ^H× 1× C_i as well as Z^w_i∈ℝ^1 × W × C_i store the average pooling results of each feature map in F^S_i across the dimensions of H and W, respectively. Z^w_i is subsequently reshaped into ℝ^W × 1 × C_i. We further apply a convolutional layer and a Sigmoid function to make full use of the captured positional information, enhancing the network's ability to accurately emphasize regions of interest. This process can be formulated as follows:

    Ẑ_i = σ( 1× 1Conv ( [Z^h_i, Z^w_i] ) ),

where [·,·] denotes the concatenation operation along the spatial dimension. Then, Ẑ_i is split into two separate tensors: Ẑ^h_i∈ℝ^H× 1× C_i and Ẑ^w_i∈ℝ^W× 1× C_i. This allows interactions between Ẑ^h_i and Ẑ^w_i from both dimensions, enhancing the emphasis on regions of interest. Ẑ^w_i is then reshaped into ℝ^1× W× C_i. Each element within the two attention maps, Ẑ^h_i and Ẑ^w_i, indicates the presence of objects of interest across respective rows or columns. Ẑ^h_i and Ẑ^w_i are applied to F^S_i to more accurately pinpoint the exact location of the object of interest, which can be written as follows:

    F^F_i = F^S_i ⊙Ẑ^h_i ⊙Ẑ^w_i.

Finally, we obtain the fused features ℱ^F={F^F_1, …,F^F_4 }, which are forwarded to the decoder to obtain the final semantic prediction. Given the outstanding performance of the multi-scale Transformer decoder employed in RoadFormer, we retain this design. Please refer to <cit.> for more details on the decoder and loss function.



§ EXPERIMENTS




 §.§ Datasets



We first compare RoadFormer+ with other SoTA scene parsing networks on the following seven RGB-X datasets:


  §.§.§ SYN-UDTIRI <cit.>



This dataset contains over 10,000 pairs of stereo road images, along with corresponding depth maps, surface normal information, and semantic annotations, including three categories: freespace, road defect, and other objects. It is created using the CARLA simulator <cit.> and first introduced in our previous work <cit.>. The input images are resized to a resolution of 640×352 pixels.



  §.§.§ KITTI Road <cit.>



This dataset has 289 pairs of stereo road images and their corresponding LiDAR point clouds for both model training and validation. We employ a data preprocessing strategy akin to that detailed in <cit.>. The input images are resized to a resolution of 1,280×384 pixels.



  §.§.§ Cityscapes <cit.>



This widely used urban scene dataset contains 2,975 stereo training images and 500 validation images, with well-annotated semantic annotations. Notably, the surface normal information is derived from depth images generated using RAFT-Stereo <cit.>, trained on the KITTI dataset <cit.>. The input images are resized to a resolution of 1,024×512 pixels. 




  §.§.§ KITTI Semantics <cit.>


This dataset contains 200 images and their corresponding semantic annotations across 19 classes. Surface normal information is derived from depth data acquired using ViTAStereo <cit.>, chosen for its superior accuracy. The input images are resized to a resolution of 1,280×384 pixels.



  §.§.§ MFNet <cit.>



This urban driving scene parsing dataset contains 1,569 synchronized pairs of RGB and thermal images at a resolution of 640×480 pixels. It includes semantic annotations across nine classes: bike, person, car, road lanes, guardrail, car stop, bump, color cone, and background.



  §.§.§ FMB <cit.>
 


This dataset contains 1,500 well-rectified RGB-Thermal image pairs (resolution: 800×600 pixels), captured in urban driving scenes under different illumination conditions. It provides semantic annotations across 14 classes.



  §.§.§ ZJU <cit.>
 

This RGB-Polarization dataset, designed for automated driving applications, containing 394 image pairs. Each pair contains four polarized images captured at different polarization angles (0^∘, 45^∘, 90^∘, and 135^∘). We use the angle of linear polarization (AoLP) algorithm to obtain the polarization information. The input images are resized to a resolution of 612×512 pixels.



 §.§ Experimental Setup and Evaluation Metrics



For the SYN-UDTIRI and other RGB-Normal datasets, we exclusively use surface normal information estimated using the D2NT algorithm <cit.> owing to its superior accuracy. This information serves as the “X” data to train our RoadFormer+. Additionally, depth, thermal, and polarization information are replicated across the channel dimension three times during data preprocessing to match the H × W × 3 dimensions of RGB images. During training, we employ the common data augmentation techniques used in semantic segmentation, including resizing, random cropping, and random flipping of RGB-X image pairs. Additionally, we make random adjustments to the brightness, contrast, saturation, and hue of the RGB images. All networks are trained for the same number of epochs on an NVIDIA RTX 3090 GPU using the AdamW optimizer <cit.>, with a polynomial decay strategy for the learning rate <cit.>. The initial learning rate is set at 10^-4 with a weight decay of 5 × 10^-2, and learning rate multipliers of 10^-1 are applied to the weight-sharing backbone.

We evaluate the performance of our models using five common metrics: accuracy (Acc), precision (Pre), recall (Rec), intersection over union (IoU), and F-score (Fsc). We refer readers to our previous work <cit.> for more details on these metrics. Additionally, the evaluation metrics used for the KITTI Road and KITTI Semantics benchmarks are available on the official webpage: <cvlibs.net/datasets/kitti>.




























 §.§ Comparison with SoTA Networks

We first conduct experiments on four RGB-Normal datasets. The quantitative results on the SYN-UDTIRI, Cityscapes, KITTI Road, and KITTI Semantics datasets are presented in Tables <ref>-<ref>, respectively. In these experiments, the symbols “B” and “L” respectively denote the use of ConvNeXt-B and ConvNeXt-L as the backbones. These results demonstrate that our proposed RoadFormer+ significantly outperforms all other SoTA networks, including our previous work RoadFormer <cit.>, across all four RGB-Normal datasets. This validates its exceptional performance and robustness in effectively parsing various types of road scenes. Notably, as shown in Tables <ref>, RoadFormer+ based on ConvNeXt-B reduces the number of learnable parameters by 65% compared to RoadFormer.

Furthermore, we conduct experiments on the Cityscapes dataset by treating it as both a binary segmentation task (road versus background) and a full-category segmentation task (19 labeled categories plus an “ignore” category). Experimental results are presented in Tables <ref> and <ref>, respectively. We also compare RoadFormer+ with four SoTA single-modal networks. It is worth noting that traditional data-fusion networks, which typically employ basic element-wise addition or feature-level concatenation for feature fusion, perform worse than single-modal networks. This underperformance may be attributed to the noise present in disparity maps used for surface normal estimation, which are derived directly from a stereo-matching network pre-trained on the KITTI dataset. Experimental results further demonstrate that RoadFormer+ effectively overcomes this issue through feature recalibration and enhancement, thus preventing performance degradation even when surface normal information is inaccurate.

We submit the test set results obtained by RoadFormer+ to both the KITTI Road and KITTI Semantics benchmarks for performance comparison. As shown in Tables <ref> and <ref>, RoadFormer+ ranks first on the KITTI Road benchmark and ranks third on the KITTI Semantics benchmark. Notably, the top-performing SoTA methods in the KITTI Semantics benchmark employ sequential frames (±10) from the scene flow subset for data augmentation. Despite this, RoadFormer+ exhibits superior performance in urban scene parsing compared to all previously published methods.

Furthermore, we explore the applicability of RoadFormer+ for RGB-Thermal and RGB-Polarization scene parsing. Experimental results on three public datasets, MFNet (RGB-Thermal), FMB (RGB-Thermal), and ZJU (RGB-Polarization), demonstrate the superiority of RoadFormer+ over other task-specific data-fusion networks for these modalities. Impressively, RoadFormer+ achieves an improvement in mIoU of 1.2-9.5% on the MFNet dataset, 3.9-22.5% on the FMB dataset, and 0.3-7.3% on the ZJU dataset, compared to other SoTA methods. These results underscore the versatility of our network in handling diverse data types. It is important to note that since the “bicycle” category is not included in the test set of the FMB dataset, we report the mIoU metrics excluding  the “bicycle” category.



Qualitative comparisons on the KITTI Road, Cityscapes, and MFNet datasets are presented in Figs. <ref>-<ref>. The dual-branch feature fusion design of RoadFormer+ enables effective capture of both local and global contexts, thereby outperforming previous single-branch heterogeneous feature fusion approaches. Our method not only demonstrates robust performance in comprehensive scene understanding but also excels in delineating detailed boundaries. Additionally, RoadFormer+ exhibits superior capabilities in handling challenging conditions such as darkness and fog, demonstrating its versatility across diverse environmental scenarios. Furthermore, RoadFormer+ consistently delivers robust performance across various illumination conditions. As illustrated in the second row of Fig <ref>, RoadFormer+ outperforms all existing data-fusion methods in handling overexposed scenes within the MFNet dataset.



 §.§ Ablation Studies

We conduct ablation studies on the SYN-UDTIRI, MFNet, and ZJU datasets. Our baseline is built upon RoadFormer <cit.>, and all implementation details are consistent with those described in Sect. <ref>.







  §.§.§ Effectiveness of HFDE


Building on our previous findings stated in <cit.> that demonstrated the effectiveness of ConvNeXt <cit.> in urban scene parsing, we continue to employ it as the backbone in this study. We investigate two backbone training strategies: weight-sharing and weight-separating. The results, presented in Table <ref>, show that the weight-sharing strategy not only achieves performance comparable to the weight-separating strategy across three RGB-X datasets but also significantly reduces the model's parameters by nearly half. This observation calls into question the utility of traditional duplex encoder designs in these applications. 

We further validate the effectiveness of our proposed GFE and LFE on the SYN-UDTIRI dataset in terms of heterogeneous feature enhancement. It is evident that using either GFEs or LFEs independently can effectively enhance our model's performance, and their combined use results in an IoU increase of 0.57%. Additionally, we compare ConvNeXt with recently proposed models, including the Transformer-based DiNAT <cit.> and UniRepLKNet <cit.>, which both employ large-kernel convolutions. The results affirm that ConvNeXt continues to exhibit superior performance compared to other backbones.









  §.§.§ Effectiveness of the MHFF Block


As illustrated in Table <ref>, we utilize RoadFormer as the baseline and alternately replace its feature fusion module with components from our proposed MHFF block to validate the efficacy of the dual-branch feature fusion design. First, we maintain RoadFormer's HFFM and FFRM to fuse global and local features, with the results depicted in the first row. As indicated in the second row, we maintain the use of the HFFM for global feature fusion while integrating the proposed LFFM for local feature fusion, resulting in performance improvements on the SYN-UDTIRI and MFNet datasets, while maintaining stability on the ZJU dataset. Subsequently, HFFM is replaced with our proposed GFRM, with results shown in the third row. Finally, FFRM is replaced with the proposed FEIM, with results presented in the fourth row. The experimental results underscore the individual effectiveness and compatibility of our proposed GFRM, LFFM, and FEIM. When fully integrated, these modules significantly enhance RoadFormer+'s performance in processing three types of RGB-X data compared to the original RoadFormer's feature fusion method. The feature fusion method presented in row four is our proposed MHFF block. To further validate the effectiveness of the channel expansion design in LFFM and the collaborative processing of Z^h_i and Z^w_i in FEIM, additional experiments are conducted. Removing these operations leads to a decline in the overall performance, as demonstrated in rows five and six.






§ CONCLUSION


This article reviewed designs for heterogeneous feature extraction and fusion strategies and introduced RoadFormer+, a highly efficient, robust, and applicable urban scene parsing network. Breaking down our contributions further, our work contains five key technical advancements: two modules for feature decoupling in the encoding stage, and three new components within the feature fusion module. The effectiveness of each contribution was validated through extensive experiments. RoadFormer+ outperforms other SoTA algorithms across multiple RGB-X scene parsing datasets. Our future work will primarily focus on investigating lightweight algorithms to enhance adaptability to terminal devices.

IEEEtran
 





