Update README.md

README.md

@@ -1,76 +1,86 @@
 ---
 model-index:
-- name: Duino-Idar
-  paper: https://huggingface.co/Duino/Duino-Idar/blob/main/README.md # Link to this README.md
-  results:
-  - task:
-      type: 3D Indoor Mapping # Task type as object
-    dataset:
-      name: Mobile Video # Dataset name as object
-      type: Video # Added dataset type here to resolve the error
-    metrics:
-      - name: Qualitative 3D Reconstruction
-        type: Visual Inspection
-        value: "Visually Inspected; Subjectively assessed for geometric accuracy and completeness of the point cloud." # Added value
-      - name: Semantic Accuracy (Conceptual)
-        type: Qualitative Assessment
-        value: "Qualitatively Assessed; Subjectively evaluated for the relevance and coherence of semantic labels generated for indoor scenes." # Added value
+  - name: Duino-Idar
+    paper: https://huggingface.co/Duino/Duino-Idar/blob/main/README.md
+    results:
+      - task:
+          type: "3D Indoor Mapping"
+        dataset:
+          name: "Mobile Video"
+          type: "Video"
+        metrics:
+          - name: "Qualitative 3D Reconstruction"
+            type: "Visual Inspection"
+            value: "Visually Inspected; Subjectively assessed for geometric accuracy and completeness of the point cloud."
+          - name: "Semantic Accuracy (Conceptual)"
+            type: "Qualitative Assessment"
+            value: "Qualitatively Assessed; Subjectively evaluated for the relevance and coherence of semantic labels generated for indoor scenes."
 language: en
 license: mit
 tags:
-- 3d-mapping
-- depth-estimation
-- semantic-segmentation
-- vision-language-model
-- indoor-scene-understanding
-- mobile-video
-- dpt
-- paligemma
-- gradio
-- point-cloud
-author: Jalal Mansour (Jalal Duino)
+  - 3d-mapping
+  - depth-estimation
+  - semantic-segmentation
+  - vision-language-model
+  - indoor-scene-understanding
+  - mobile-video
+  - dpt
+  - paligemma
+  - gradio
+  - point-cloud
+author: "Jalal Mansour (Jalal Duino)"
 date_created: 2025-02-18
 email: [email protected]
 hf_hub_url: https://huggingface.co/Duino/Duino-Idar
 ---
 
-# Duino-Idar: An Interactive Indoor 3D Mapping System via Mobile Video with Semantic Enrichment
 
-**Abstract**
+---
+
+# ***Duino-Idar: An Interactive Indoor 3D Mapping System via Mobile Video with Semantic Enrichment*** #
 
-This paper introduces Duino-Idar, a novel end-to-end system for generating interactive 3D maps of indoor environments from mobile video. Leveraging state-of-the-art monocular depth estimation techniques, specifically DPT (Dense Prediction Transformer)-based models, and semantic understanding via a fine-tuned vision-language model (PaLiGemma), Duino-Idar offers a comprehensive solution for indoor scene reconstruction. The system extracts key frames from video input, computes depth maps, constructs a 3D point cloud, and enriches it with semantic labels. A user-friendly Gradio-based graphical user interface (GUI) facilitates video upload, processing, and interactive 3D scene exploration. This research details the system's architecture, implementation, and potential applications in areas such as indoor navigation, augmented reality, and automated scene understanding, setting the stage for future enhancements including LiDAR integration for improved accuracy and robustness.
+ **Abstract**
 
-**Keywords:** 3D Mapping, Indoor Reconstruction, Mobile Video, Depth Estimation, Semantic Segmentation, Vision-Language Models, DPT, PaLiGemma, Point Cloud, Gradio, Interactive Visualization.
+> This paper introduces **Duino-Idar**, a novel end-to-end system for generating interactive 3D maps of indoor environments using mobile video. By leveraging state-of-the-art monocular depth estimation (via DPT-based models) alongside semantic understanding from a fine-tuned vision-language model (PaLiGemma), Duino-Idar provides a comprehensive solution for indoor scene reconstruction. The system extracts key frames from video input, computes depth maps, builds a 3D point cloud, and enriches it with semantic labels. A user-friendly Gradio-based GUI allows video upload, processing, and interactive exploration of the 3D scene. This research details the system's architecture, implementation, and potential applications in indoor navigation, augmented reality, and automated scene understanding, and outlines future improvements including LiDAR integration for enhanced accuracy.
+
+**Keywords:**  
+3D Mapping, Indoor Reconstruction, Mobile Video, Depth Estimation, Semantic Segmentation, Vision-Language Models, DPT, PaLiGemma, Point Cloud, Gradio, Interactive Visualization
 
 ---
 
 ## 1. Introduction
 
-Recent advancements in computer vision and deep learning have significantly propelled the field of 3D scene reconstruction from 2D imagery. Mobile devices, now ubiquitous and equipped with high-quality cameras, provide a readily available source of video data suitable for spatial mapping. While monocular depth estimation has matured considerably, enabling real-time applications, many existing 3D reconstruction approaches lack a crucial component: semantic understanding of the scene. This semantic context is vital for enabling truly interactive and context-aware applications, such as augmented reality (AR) navigation, object recognition, and scene understanding for robotic systems.
+Advances in computer vision and deep learning have transformed 3D scene reconstruction from 2D images. With the ubiquity of mobile devices equipped with high-quality cameras, mobile video offers an accessible data source for spatial mapping. While monocular depth estimation techniques have matured for real-time applications, many 3D reconstruction approaches still lack semantic context—a critical component for applications such as augmented reality navigation, object recognition, and robotic scene understanding.
 
-To address this gap, we present Duino-Idar, an innovative system that integrates a robust depth estimation pipeline with a fine-tuned vision-language model, PaLiGemma, to enhance indoor 3D mapping. The system's name, Duino-Idar, reflects the vision of combining accessible technology ("Duino," referencing approachability and user-centric design) with advanced spatial sensing ("Idar," hinting at the potential for LiDAR integration in future iterations, although the current prototype focuses on vision-based depth). This synergistic combination not only achieves geometric reconstruction but also provides semantic enrichment, significantly enhancing both visualization and user interaction capabilities. This paper details the architecture, implementation, and potential of Duino-Idar, highlighting its contribution to accessible and semantically rich indoor 3D mapping.
+**Duino-Idar** addresses this gap by combining a robust depth estimation pipeline with a fine-tuned vision-language model, PaLiGemma, to enhance indoor 3D mapping. The name "Duino-Idar" reflects the fusion of user-friendly technology ("Duino") with advanced spatial sensing ("Idar"), hinting at future LiDAR integration while currently focusing on vision-based depth estimation. This paper presents the system architecture, implementation details, and potential use cases of Duino-Idar, demonstrating its contribution toward accessible and semantically enriched indoor mapping.
 
 ---
 
 ## 2. Related Work
 
-Our work builds upon and integrates several key areas of research:
+Our work builds upon three key research areas:
+
+### 2.1 Monocular Depth Estimation
 
-### 2.1 Monocular Depth Estimation:
+Monocular depth estimation forms the backbone of our geometric reconstruction. Pioneering works such as MiDaS [1] and DPT [2] have shown impressive capabilities in inferring depth from single images. In particular, DPT utilizes transformer architectures to capture global context, significantly enhancing depth accuracy compared to earlier CNN-based methods. The depth normalization process in DPT-like models is illustrated in Equation (1):
 
-The foundation of our geometric reconstruction lies in monocular depth estimation. Models such as MiDaS [1] and DPT [2] have demonstrated remarkable capabilities in inferring depth from single images. DPT, in particular, leverages transformer architectures to capture global contextual information, leading to improved depth accuracy compared to earlier convolutional neural network (CNN)-based methods. Equation (1) illustrates the depth normalization process used in DPT-like models to scale the predicted depth map to a usable range.
+$$
+D = f(I; \theta)
+$$
 
-### 2.2 3D Reconstruction Techniques:
+*where \( D \) is the depth map estimated from the image \( I \) using model parameters \( \theta \).*
 
-Generating 3D point clouds or meshes from 2D inputs is a well-established field, encompassing techniques from photogrammetry [3] and Simultaneous Localization and Mapping (SLAM) [4]. Our approach utilizes depth maps derived from DPT to construct a point cloud, offering a simpler yet effective method for 3D scene representation, particularly suitable for indoor environments where texture and feature richness can support monocular depth estimation. The transformation from 2D pixel coordinates to 3D space is mathematically described by the pinhole camera model, as shown in Equations (2)-(4).
+### 2.2 3D Reconstruction Techniques
 
-### 2.3 Vision-Language Models for Semantic Understanding:
+Reconstructing 3D point clouds or meshes from 2D inputs is a well-established field, encompassing methods from photogrammetry [3] and SLAM [4]. Duino-Idar leverages depth maps from the DPT model to create point clouds using the pinhole camera model. Equations (2)–(4) detail the transformation from 2D pixel coordinates to 3D space.
 
-Vision-language models (VLMs) have emerged as powerful tools for bridging the gap between visual and textual understanding. PaLiGemma [5] is a state-of-the-art multimodal model that integrates image understanding with natural language processing. Fine-tuning such models on domain-specific datasets, such as indoor scenes, allows for the generation of semantic annotations and descriptions that can be overlaid on reconstructed 3D models, enriching them with contextual information. The fine-tuning process for PaLiGemma, aimed at minimizing the token prediction loss, is formalized in Equation (5).
+### 2.3 Vision-Language Models for Semantic Understanding
 
-### 2.4 Interactive 3D Visualization:
+Vision-language models (VLMs) bridge visual data and textual descriptions. PaLiGemma [5] is a state-of-the-art multimodal model that integrates image interpretation with natural language processing. Fine-tuning on indoor scene datasets allows the model to generate meaningful semantic labels that are overlaid on the reconstructed 3D models.
 
-Effective visualization is crucial for user interaction with 3D data. Libraries like Open3D [6] and Plotly [7] provide tools for interactive exploration of 3D point clouds and meshes. Open3D, in particular, offers robust functionalities for point cloud manipulation, rendering, and visualization, making it an ideal choice for desktop-based interactive 3D scene exploration. For web-based interaction, Plotly offers excellent capabilities for embedding interactive 3D visualizations within web applications.
+### 2.4 Interactive 3D Visualization
+
+> Interactive visualization is key for effective 3D data exploration. Libraries such as Open3D [6] and Plotly [7] enable users to interact with 3D point clouds through rotation, zooming, and panning. Open3D is ideal for desktop-based exploration, while Plotly supports web-based interactive 3D visualizations.
 
 ---
 
@@ -78,58 +88,77 @@ Effective visualization is crucial for user interaction with 3D data. Libraries
 
 ### 3.1 Overview
 
-The Duino-Idar system is structured into three primary modules, as illustrated in Figure 1:
-
-1.  **Video Processing and Frame Extraction:** This module ingests mobile video input and extracts representative key frames at configurable intervals to reduce computational redundancy and capture scene changes effectively.
-2.  **Depth Estimation and 3D Reconstruction:** Each extracted frame is processed by a DPT-based depth estimator to generate a depth map. These depth maps are then converted into 3D point clouds using a pinhole camera model, transforming 2D pixel coordinates into 3D spatial positions.
-3.  **Semantic Enrichment and Visualization:** A fine-tuned PaLiGemma model provides semantic annotations for the extracted key frames, enriching the 3D reconstruction with object labels and scene descriptions. A Gradio-based GUI integrates these modules, providing a user-friendly interface for video upload, processing, interactive 3D visualization, and exploration of the semantically enhanced 3D scene.
-
-**Figure 1: System Architecture Diagram**
-
-```mermaid
-graph LR
-    A[Mobile Video Input] --> B(Video Processing & Frame Extraction);
-    B --> C(Depth Estimation (DPT));
-    C --> D(3D Reconstruction (Pinhole Model));
-    B --> E(Semantic Enrichment (PaLiGemma));
-    D --> F(Point Cloud);
-    E --> G(Semantic Labels);
-    F & G --> H(Semantic Integration);
-    H --> I(Interactive 3D Viewer (Open3D/Plotly));
-    I --> J[Gradio GUI & User Interaction];
-    style B fill:#f9f,stroke:#333,stroke-width:2px
-    style C fill:#ccf,stroke:#333,stroke-width:2px
-    style D fill:#fcc,stroke:#333,stroke-width:2px
-    style E fill:#cfc,stroke:#333,stroke-width:2px
-    style H fill:#eee,stroke:#333,stroke-width:2px
-    style I fill:#ace,stroke:#333,stroke-width:2px
-```
-*Figure 1: Duino-Idar System Architecture. The diagram illustrates the flow of data through the system modules, from video input to interactive 3D visualization with semantic enrichment.*
+The Duino-Idar system comprises three main modules, as depicted in **Figure 1**:
 
-### 3.2 Detailed Pipeline
+1. **Video Processing and Frame Extraction:**  
+   Ingests mobile video and extracts key frames at configurable intervals to capture scene changes while reducing redundancy.
 
-The Duino-Idar pipeline operates through the following detailed steps:
+2. **Depth Estimation and 3D Reconstruction:**  
+   Processes each extracted frame using a DPT-based depth estimator to generate depth maps. These maps are then converted into 3D point clouds via the pinhole camera model.
 
-1.  **Input Module:**
-    *   **Video Upload:** Users initiate the process by uploading a mobile-recorded video via the Gradio web interface.
-    *   **Frame Extraction:** OpenCV is employed to extract frames from the uploaded video at regular, user-configurable intervals. This interval determines the density of key frames used for reconstruction, balancing computational cost with scene detail.
+3. **Semantic Enrichment and Visualization:**  
+   Utilizes a fine-tuned PaLiGemma model to produce semantic annotations for each key frame, enriching the 3D reconstruction with object labels and scene descriptions. The Gradio-based GUI integrates these modules for an interactive user experience.
 
-2.  **Depth Estimation Module:**
-    *   **Preprocessing:** Each extracted frame undergoes preprocessing, including resizing and normalization, to optimize it for input to the DPT model. This ensures consistent input dimensions and value ranges for the depth estimation network.
-    *   **Depth Prediction:** The preprocessed frame is fed into the DPT model, which generates a depth map. This depth map represents the estimated distance of each pixel in the image from the camera.
-    *   **Normalization and Scaling:** The raw depth map is normalized to a standard range (e.g., 0-1 or 0-255) for subsequent 3D reconstruction and visualization. Equation (1) details the normalization process.
-
-3.  **3D Reconstruction Module:**
-    *   **Point Cloud Generation:** A pinhole camera model is applied to convert the depth map and corresponding pixel coordinates into 3D coordinates in camera space. Color information from the original frame is associated with each 3D point to create a colored point cloud. Equations (2), (3), and (4) formalize this transformation.
-    *   **Point Cloud Aggregation:** To build a comprehensive 3D model, point clouds generated from multiple key frames are aggregated. In this initial implementation, we assume a static camera or negligible inter-frame motion for simplicity. More advanced implementations could incorporate camera pose estimation and point cloud registration for improved accuracy, especially in dynamic scenes. The aggregation process is mathematically represented by Equation (4).
-
-4.  **Semantic Enhancement Module:**
-    *   **Vision-Language Processing:** The fine-tuned PaLiGemma model processes the key frames to generate scene descriptions and semantic labels. The model is prompted to identify objects and provide contextual information relevant to indoor scenes.
-    *   **Semantic Data Integration:** Semantic labels generated by PaLiGemma are overlaid onto the reconstructed point cloud. This integration can be achieved through various methods, such as associating semantic labels with clusters of points or generating bounding boxes around semantically labeled objects within the 3D scene.
+### 3.2 Detailed Pipeline
 
-5.  **Visualization and User Interface Module:**
-    *   **Interactive 3D Viewer:** The final semantically enriched 3D model is visualized using Open3D (or Plotly for web-based deployments). Users can interact with the 3D scene, rotating, zooming, and panning to explore the reconstructed environment.
-    *   **Gradio GUI:** A user-friendly Gradio web interface provides a seamless experience, allowing users to upload videos, initiate the processing pipeline, and interactively navigate the resulting 3D scene. The GUI also provides controls for adjusting parameters like frame extraction interval and potentially visualizing semantic labels.
+1. **Input Module:**
+   - **Video Upload:** Users upload a mobile-recorded video via the Gradio interface.
+   - **Frame Extraction:** OpenCV extracts frames at user-defined intervals, balancing detail with computational cost.
+
+2. **Depth Estimation Module:**
+   - **Preprocessing:** Frames are resized and normalized before being fed into the DPT model.
+   - **Depth Prediction:** The DPT model generates a depth map for each frame.
+   - **Normalization and Scaling:**  
+     The raw depth map is normalized and scaled for visualization:
+     
+     $$
+     D_{\text{norm}}(u,v) = \frac{D(u,v)}{\max_{(u,v)} D(u,v)}
+     $$
+     
+     and, assuming a maximum depth \( Z_{\max} \):
+     
+     $$
+     z(u,v) = D_{\text{norm}}(u,v) \times Z_{\max}
+     $$
+
+3. **3D Reconstruction Module:**
+   - **Point Cloud Generation:**  
+     Using the pinhole camera model, each pixel is mapped to 3D space:
+     
+     $$
+     x = \frac{(u - c_x) \cdot z(u,v)}{f_x}, \quad y = \frac{(v - c_y) \cdot z(u,v)}{f_y}, \quad z = z(u,v)
+     $$
+     
+     In matrix form:
+     
+     $$
+     \begin{pmatrix} x \\ y \\ z \end{pmatrix} = z(u,v) \cdot K^{-1} \begin{pmatrix} u \\ v \\ 1 \end{pmatrix}
+     $$
+     
+   - **Point Cloud Aggregation:**  
+     Point clouds from multiple key frames are aggregated to form the final 3D model:
+     
+     $$
+     P = \bigcup_{i=1}^{M} P_i
+     $$
+
+4. **Semantic Enhancement Module:**
+   - **Vision-Language Processing:**  
+     PaLiGemma processes key frames to generate scene descriptions and semantic labels.
+   - **Semantic Data Integration:**  
+     These labels are overlaid on the point cloud to provide contextual scene information.
+
+5. **Visualization and User Interface Module:**
+   - **Interactive 3D Viewer:**  
+     The enriched 3D model is rendered using Open3D or Plotly, allowing interactive exploration.
+   - **Gradio GUI:**  
+     A user-friendly web interface supports video upload, pipeline execution, and 3D scene visualization.
+
+**Figure 1: Duino-Idar System Architecture Diagram**
+
+
+
+*Figure 1: The flow from mobile video input to interactive 3D visualization with semantic enrichment.*
 
 ---
 
@@ -137,94 +166,95 @@ The Duino-Idar pipeline operates through the following detailed steps:
 
 ### 4.1 Mathematical Framework
 
-The Duino-Idar system relies on several core mathematical principles:
+**1. Depth Estimation via Deep Network**
 
-**1. Depth Estimation via Deep Network:**
+Let \( I \in \mathbb{R}^{H \times W \times 3} \) be the input image. The DPT model \( f \) estimates the depth map \( D \):
 
-Let $I \in \mathbb{R}^{H \times W \times 3}$ represent the input image of height $H$ and width $W$. The DPT model, denoted as $f$, with learnable parameters $\theta$, estimates the depth map $D$:
+$$
+D = f(I; \theta) \quad \text{(1)}
+$$
 
-**(1)**  $D = f(I; \theta)$
+Normalize the depth map:
 
-The depth map $D$ is then normalized to obtain $D_{\text{norm}}$:
+$$
+D_{\text{norm}}(u,v) = \frac{D(u,v)}{\max_{(u,v)} D(u,v)} \quad \text{(2)}
+$$
 
-**(2)**  $D_{\text{norm}}(u,v) = \frac{D(u,v)}{\displaystyle \max_{(u,v)} D(u,v)}$
+Scale with maximum depth \( Z_{\max} \):
 
-If a maximum physical depth $Z_{\max}$ is assumed, the scaled depth $z(u,v)$ is:
+$$
+z(u,v) = D_{\text{norm}}(u,v) \times Z_{\max} \quad \text{(3)}
+$$
 
-**(3)**  $z(u,v) = D_{\text{norm}}(u,v) \times Z_{\max}$
+For 8-bit scaling:
 
-For practical implementation and visualization, we often scale the depth to an 8-bit range:
+$$
+D_{\text{scaled}}(u,v) = \frac{D(u,v)}{\max_{(u,v)} D(u,v)} \times 255 \quad \text{(4)}
+$$
 
-**(4)**  $D_{\text{scaled}}(u,v) = \frac{D(u,v)}{\displaystyle \max_{(u,v)} D(u,v)} \times 255$
+**2. 3D Reconstruction using the Pinhole Camera Model**
 
-**2. 3D Reconstruction with Pinhole Camera Model:**
+With intrinsic parameters \( f_x, f_y \) and principal point \( (c_x, c_y) \), the intrinsic matrix is:
 
-Assuming a pinhole camera model with intrinsic parameters: focal lengths $(f_x, f_y)$ and principal point $(c_x, c_y)$, the intrinsic matrix $K$ is:
-
-**(5)**  $K = \begin{pmatrix}
+$$
+K = \begin{pmatrix}
 f_x & 0 & c_x \\
 0 & f_y & c_y \\
 0 & 0 & 1
-\end{pmatrix}$
-
-Given a pixel $(u, v)$ and its depth value $z(u,v)$, the 3D coordinates $(x, y, z)$ in the camera coordinate system are:
-
-**(6)**  $x = \frac{(u - c_x) \cdot z(u,v)}{f_x}$
+\end{pmatrix} \quad \text{(5)}
+$$
 
-**(7)**  $y = \frac{(v - c_y) \cdot z(u,v)}{f_y}$
+Given pixel \( (u,v) \) and depth \( z(u,v) \), compute 3D coordinates:
 
-**(8)**  $z = z(u,v)$
+$$
+x = \frac{(u - c_x) \cdot z(u,v)}{f_x}, \quad y = \frac{(v - c_y) \cdot z(u,v)}{f_y}, \quad z = z(u,v) \quad \text{(6), (7), (8)}
+$$
 
-In matrix form:
+Or in matrix form:
 
-**(9)**  $\begin{pmatrix}
-x \\
-y \\
-z
-\end{pmatrix}
-= z(u,v) \cdot K^{-1} \begin{pmatrix}
-u \\
-v \\
-1
-\end{pmatrix}$
+$$
+\begin{pmatrix}
+x \\ y \\ z
+\end{pmatrix} = z(u,v) \cdot K^{-1} \begin{pmatrix}
+u \\ v \\ 1
+\end{pmatrix} \quad \text{(9)}
+$$
 
-**3. Aggregation of Multiple Frames:**
+**3. Aggregation of Multiple Frames**
 
-Let $P_i$ be the point cloud from the $i^{th}$ frame, where $P_i = \{(x_{i,j}, y_{i,j}, z_{i,j}) \mid j = 1, 2, \ldots, N_i\}$. The overall point cloud $P$ is the union:
+For point cloud \( P_i \) from the \( i^\text{th} \) frame:
 
-**(10)** $P = \bigcup_{i=1}^{M} P_i$
+$$
+P = \bigcup_{i=1}^{M} P_i \quad \text{(10)}
+$$
 
-where $M$ is the number of frames.
+**4. Fine-Tuning PaLiGemma Loss**
 
-**4. Fine-Tuning PaLiGemma Loss:**
+For an image \( I \) and caption tokens \( c = (c_1, c_2, \ldots, c_T) \), minimize the cross-entropy loss:
 
-For fine-tuning PaLiGemma, given an image $I$ and caption tokens $c = (c_1, c_2, \ldots, c_T)$, the cross-entropy loss $\mathcal{L}$ is minimized:
-
-**(11)** $\mathcal{L} = -\sum_{t=1}^{T} \log P(c_t \mid c_{<t}, I)$
-
-where $P(c_t \mid c_{<t}, I)$ is the conditional probability of predicting the $t^{th}$ token given the preceding tokens $c_{<t}$ and the input image $I$.
+$$
+\mathcal{L} = -\sum_{t=1}^{T} \log P(c_t \mid c_{<t}, I) \quad \text{(11)}
+$$
 
 ### 4.2 Implementation Environment and Dependencies
 
-The Duino-Idar system is implemented in Python, leveraging the following libraries:
+Duino-Idar is implemented in Python using the following libraries:
 
-*   **Deep Learning:** `Transformers`, `PEFT`, `BitsAndBytes`, `Torch`, `Torchvision`, `DPT (Dense Prediction Transformer)`
-*   **Computer Vision:** `OpenCV-Python`, `Pillow`
-*   **3D Visualization:** `Open3D`, `Plotly` (optional for web)
-*   **GUI:** `Gradio`
-*   **Data Manipulation:** `NumPy`
+- **Deep Learning:** `transformers`, `peft`, `bitsandbytes`, `torch`, `torchvision`, and the `DPT` model.
+- **Computer Vision:** `opencv-python`, `Pillow`
+- **3D Visualization:** `open3d`, `plotly` (for web deployments)
+- **GUI:** `gradio`
+- **Data Manipulation:** `numpy`
 
-Installation can be achieved using `pip`:
+Install the dependencies using:
 
 ```bash
 pip install transformers peft bitsandbytes gradio opencv-python pillow numpy torch torchvision torchaudio open3d
 ```
 
-### 4.3 Code Snippets and Dynamicity
+### 4.3 Code Snippets
 
-Here are illustrative code snippets demonstrating key functionalities. These are excerpts from the provided datasheet code and are used for demonstration purposes within this paper.
-
-#### 4.3.1 Depth Estimation using DPT:
+#### 4.3.1 Depth Estimation using DPT
 
 ```python
 import torch
@@ -239,18 +269,15 @@ def estimate_depth(image):
     inputs = feature_extractor(images=image, return_tensors="pt")
     with torch.no_grad():
         depth_map = dpt_model(**inputs).predicted_depth.squeeze().numpy()
-    depth_map = (depth_map / np.max(depth_map) * 255).astype(np.uint8) # Normalize
+    depth_map = (depth_map / np.max(depth_map) * 255).astype(np.uint8)  # Normalize to 8-bit
     return depth_map
 
 # Example usage:
-image = Image.open("example_frame.jpg") # Replace with an actual image path
+image = Image.open("example_frame.jpg")  # Replace with an actual image path
 depth_map = estimate_depth(image)
-# depth_map is now a NumPy array representing the estimated depth
 ```
 
-This code segment demonstrates the dynamic process of depth estimation. Given an input image, the `estimate_depth` function dynamically processes it through the pre-trained DPT model to produce a depth map. The normalization step ensures the output depth map is within a usable range for subsequent processing.
-
-#### 4.3.2 3D Point Cloud Reconstruction:
+#### 4.3.2 3D Point Cloud Reconstruction
 
 ```python
 import open3d as o3d
@@ -258,33 +285,31 @@ import numpy as np
 
 def reconstruct_3d(depth_map, image):
     h, w = depth_map.shape
-    fx = fy = max(h, w) / 2.0 # Approximate intrinsics
+    fx = fy = max(h, w) / 2.0  # Approximate focal lengths
     cx, cy = w / 2.0, h / 2.0
     points = []
     colors = []
-    image_np = np.array(image) / 255.0 # Normalize image to 0-1
+    image_np = np.array(image) / 255.0  # Normalize image
 
     for v in range(h):
         for u in range(w):
-            z = depth_map[v, u] / 255.0 * 5.0 # Scaled depth
+            z = depth_map[v, u] / 255.0 * 5.0  # Scale depth
             x = (u - cx) * z / fx
             y = (v - cy) * z / fy
             points.append([x, y, z])
-            colors.append(image_np[v, u]) # Color from original image
+            colors.append(image_np[v, u])  # RGB color from image
 
     pcd = o3d.geometry.PointCloud()
     pcd.points = o3d.utility.Vector3dVector(np.array(points))
     pcd.colors = o3d.utility.Vector3dVector(np.array(colors))
     return pcd
 
-# Example usage (assuming depth_map and image are already loaded):
+# Example usage:
 point_cloud = reconstruct_3d(depth_map, image)
-o3d.io.write_point_cloud("output.ply", point_cloud) # Save point cloud
+o3d.io.write_point_cloud("output.ply", point_cloud)
 ```
 
-This snippet shows the dynamic reconstruction of a 3D point cloud from a depth map and a corresponding image. The function dynamically iterates through each pixel, calculates its 3D coordinates based on the depth value and pinhole camera model, and associates color information. The resulting point cloud is then saved in PLY format, ready for visualization.
-
-#### 4.3.3 Gradio Interface for Interactive Visualization:
+#### 4.3.3 Gradio Interface for Interactive Visualization
 
 ```python
 import gradio as gr
@@ -292,23 +317,9 @@ import open3d as o3d
 
 def visualize_3d_model(ply_file):
     pcd = o3d.io.read_point_cloud(ply_file)
-    o3d.visualization.draw_geometries([pcd]) # Interactive window
+    o3d.visualization.draw_geometries([pcd])
 
-def process_video(video_path):
-    """ Process video: extract frames, estimate depth, and generate a 3D model. """
-    frames = extract_frames(video_path) # Assuming extract_frames function is defined (from full code)
-    depth_maps = [estimate_depth(frame) for frame in frames]
-    final_pcd = None
-    for frame, depth_map in zip(frames, depth_maps):
-        pcd = reconstruct_3d(depth_map, frame)
-        if final_pcd is None:
-            final_pcd = pcd
-        else:
-            final_pcd += pcd
-    o3d.io.write_point_cloud("output.ply", final_pcd)
-    return "output.ply"
-
-def extract_frames(video_path, interval=10): # Example extract_frames function
+def extract_frames(video_path, interval=10):
     import cv2
     from PIL import Image
     cap = cv2.VideoCapture(video_path)
@@ -325,6 +336,18 @@ def extract_frames(video_path, interval=10): # Example extract_frames function
     cap.release()
     return frames
 
+def process_video(video_path):
+    frames = extract_frames(video_path)
+    depth_maps = [estimate_depth(frame) for frame in frames]
+    final_pcd = None
+    for frame, depth_map in zip(frames, depth_maps):
+        pcd = reconstruct_3d(depth_map, frame)
+        if final_pcd is None:
+            final_pcd = pcd
+        else:
+            final_pcd += pcd
+    o3d.io.write_point_cloud("output.ply", final_pcd)
+    return "output.ply"
 
 with gr.Blocks() as demo:
     gr.Markdown("### Duino-Idar 3D Mapping")
@@ -337,25 +360,26 @@ with gr.Blocks() as demo:
     view_btn = gr.Button("View 3D Model")
     view_btn.click(fn=visualize_3d_model, inputs=output_file, outputs=None)
 
-
 demo.launch()
 ```
 
-**Figure 2: Example Gradio Interface Screenshot - Conceptual**
-
-*(A conceptual screenshot of a Gradio interface showing a video upload area, a process button, and potentially a placeholder for 3D visualization or a link to view the 3D model. Due to limitations of text-based output, a real screenshot cannot be embedded here, but imagine a simple, functional web interface based on Gradio.)*
-
-*Figure 2: Conceptual Gradio Interface for Duino-Idar. This illustrates a user-friendly web interface for video input, processing initiation, and 3D model visualization access.*
+**Figure 2: Conceptual Gradio Interface Screenshot**  
+*(Imagine a simple web interface with a video upload area, process button, and a section to view the generated 3D model.)*
 
 ---
 
 ## 5. Experimental Setup and Demonstration
 
-While this paper focuses on system design and implementation, a preliminary demonstration was conducted to validate the Duino-Idar pipeline. Mobile videos of indoor environments (e.g., living rooms, kitchens, offices) were captured using a standard smartphone camera. These videos were then uploaded to the Duino-Idar Gradio interface.
+Preliminary tests were conducted using mobile videos of indoor scenes (e.g., living rooms, kitchens, offices). Videos were uploaded via the Gradio interface, and the pipeline executed the following steps:
 
-To illustrate the depth estimation performance conceptually, consider a simplified representation of depth accuracy across different distances.  **Note:** Markdown cannot render dynamic graphs. The following is a *text-based* approximation. For actual graphs, you would embed image links below.
+1. **Frame Extraction:** Key frames were extracted at a configurable interval.
+2. **Depth Estimation:** The DPT model generated depth maps for each frame.
+3. **3D Reconstruction:** Depth maps were transformed into colored 3D point clouds.
+4. **Semantic Labeling:** The PaLiGemma model provided semantic labels (e.g., "sofa," "table," "chair") which can later be integrated into the 3D scene.
 
-**Conceptual Depth Accuracy vs. Distance (Text-Based Graph):**
+### Conceptual Graph: Depth Accuracy vs. Distance
+
+Below is a text-based representation of the qualitative depth accuracy across different distances:
 
 ```
 Depth Accuracy (Qualitative)
@@ -370,52 +394,49 @@ Depth Accuracy (Qualitative)
 +---------------------> Distance from Camera (meters)
 ```
 
-*This is a highly simplified, qualitative representation.  For quantitative evaluation, you would typically use metrics like Root Mean Squared Error (RMSE) or Mean Absolute Error (MAE) on a dataset with ground truth depth.*
-
-The system successfully processed these videos, extracting key frames, estimating depth maps using the DPT model, and reconstructing 3D point clouds. The fine-tuned PaLiGemma model provided semantic labels, such as "sofa," "table," "chair," and "window," which were (in a conceptual demonstration, as full integration is ongoing) intended to be overlaid onto the 3D point cloud, enabling interactive semantic exploration.
-
-**Figure 3: Example 3D Point Cloud Visualization - Conceptual**
+*For quantitative evaluation, metrics like RMSE or MAE can be used on ground-truth datasets.*
 
-*(A conceptual visualization of a 3D point cloud generated by Duino-Idar, potentially showing a simple room scene with furniture. Due to limitations of text-based output, a real 3D rendering cannot be embedded here, but imagine a sparse but recognizable 3D point cloud of a room.)*
+**Figure 3: Example 3D Point Cloud Visualization (Conceptual)**  
+*Imagine a sparse yet recognizable 3D point cloud representing an indoor scene.*
 
-*Figure 3: Conceptual 3D Point Cloud Visualization. This illustrates a representative point cloud output from Duino-Idar, showing the geometric reconstruction of an indoor scene.*
+**Figure 4: Semantic Labeling Performance (Conceptual)**  
+[![Semantic Labeling Performance](link-to-your-semantic-labeling-performance-graph.png)](link-to-your-semantic-labeling-performance-graph.png)  
+*Replace the image link with an actual graph image URL to show performance per object category.*
 
-**Figure 4: Semantic Labeling Performance (Conceptual - Image Link)**
-
-[![Semantic Labeling Performance](link-to-your-semantic-labeling-performance-graph.png)](link-to-your-semantic-labeling-performance-graph.png)
-*Figure 4: Conceptual Semantic Labeling Performance.  [Replace `link-to-your-semantic-labeling-performance-graph.png`](link-to-your-semantic-labeling-performance-graph.png) with the actual URL or relative path to an image of a graph illustrating semantic labeling quality, if available. This could be a bar chart showing accuracy per object category, for instance.*
+---
 
 ## 6. Discussion and Future Work
 
-Duino-Idar demonstrates a promising approach to accessible and semantically rich indoor 3D mapping using mobile video. The integration of DPT-based depth estimation and PaLiGemma for semantic enrichment provides a valuable combination, offering both geometric and contextual understanding of indoor scenes. The Gradio interface significantly enhances usability, making the system accessible to users with varying technical backgrounds.
+Duino-Idar demonstrates a promising approach to accessible, semantically enriched indoor 3D mapping using mobile video. By integrating DPT-based depth estimation with a fine-tuned PaLiGemma for semantic context, the system provides both geometric and contextual scene understanding. The Gradio interface further democratizes access, enabling non-expert users to explore 3D reconstructions.
 
-However, several areas warrant further investigation and development:
+Future work will focus on:
+- **Enhanced Semantic Integration:** Direct overlay of semantic labels onto the point cloud via segmentation techniques.
+- **Multi-Frame Fusion & SLAM:** Incorporating robust SLAM methods to handle camera motion and improve reconstruction fidelity.
+- **LiDAR Integration:** Combining LiDAR data with vision-based depth estimation for improved robustness.
+- **Real-Time Processing:** Optimizing the pipeline (e.g., via TensorRT or mobile GPU acceleration) for near-real-time performance.
+- **Improved User Interaction:** Enhancing the Gradio interface or integrating web-based 3D viewers (e.g., Three.js) for immersive interaction.
+- **Handling Dynamic Objects:** Addressing the challenges of moving objects in indoor environments.
 
-*   **Enhanced Semantic Integration:** Future work will focus on robustly overlaying semantic labels directly onto the point cloud, potentially using point cloud segmentation techniques to associate labels with specific object regions. This will enable object-level annotation and more granular scene understanding.
-*   **Multi-Frame Fusion and SLAM:** The current point cloud aggregation is simplistic. Integrating a robust SLAM or multi-view stereo method is crucial for handling camera motion and improving reconstruction fidelity, particularly in larger or more complex indoor environments. This would also address potential drift and inconsistencies arising from independent frame processing.
-*   **LiDAR Integration (Duino-*Idar* Vision):** To truly realize the "Idar" aspect of Duino-Idar, future iterations will explore the integration of LiDAR sensors. LiDAR data can provide highly accurate depth measurements, complementing and potentially enhancing the video-based depth estimation, especially in challenging lighting conditions or for textureless surfaces. A hybrid approach combining LiDAR and vision could significantly improve the robustness and accuracy of the system.
-*   **Real-Time Processing and Optimization:** The current implementation is primarily offline. Optimizations, such as using TensorRT or mobile GPU acceleration, are necessary to achieve real-time or near-real-time mapping capabilities, making Duino-Idar suitable for applications like real-time AR navigation.
-*   **Improved User Interaction:** Further enhancements to the Gradio interface, or integration with web-based 3D viewers like Three.js, can create a more immersive and intuitive user experience, potentially enabling virtual walkthroughs and interactive object manipulation within the reconstructed 3D scene.
-*   **Handling Dynamic Objects:** The current system assumes static scenes. Future research should address the challenge of dynamic objects (e.g., people, moving furniture) within indoor environments, potentially using techniques for object tracking and removal or separate reconstruction of static and dynamic elements.
+---
 
 ## 7. Conclusion
 
-Duino-Idar presents a novel and accessible system for indoor 3D mapping from mobile video, enriched with semantic understanding through the integration of deep learning-based depth estimation and vision-language models. By leveraging state-of-the-art DPT models and fine-tuning PaLiGemma for indoor scene semantics, the system achieves both geometric reconstruction and valuable scene context. The user-friendly Gradio interface lowers the barrier to entry, enabling broader accessibility for users to create and explore 3D representations of indoor spaces. While this initial prototype lays a strong foundation, future iterations will focus on enhancing semantic integration, improving reconstruction robustness through multi-frame fusion and LiDAR integration, and optimizing for real-time performance, ultimately expanding the applicability and user experience of Duino-Idar in diverse domains such as augmented reality, robotics, and interior design.
+Duino-Idar presents a novel and accessible system for indoor 3D mapping using mobile video, enriched with semantic context. By combining cutting-edge DPT depth estimation with a fine-tuned vision-language model, the system achieves robust geometric reconstruction and scene understanding. The user-friendly Gradio interface further lowers the barrier to entry. While this prototype lays a strong foundation, future iterations will enhance semantic integration, adopt advanced multi-frame fusion techniques, integrate LiDAR data, and target real-time performance improvements. These advances will expand Duino-Idar's applicability in fields such as augmented reality, robotics, and interior design.
 
 ---
 
 ## References
 
-[1] Ranftl, R., Lasinger, K., Schindler, K., & Pollefeys, M. (2019). Towards robust monocular depth estimation: Exploiting large-scale datasets and ensembling. *arXiv preprint arXiv:1906.01591*.
+1. Ranftl, R., Lasinger, K., Schindler, K., & Pollefeys, M. (2019). *Towards robust monocular depth estimation: Exploiting large-scale datasets and ensembling*. [arXiv:1906.01591](https://arxiv.org/abs/1906.01591).
 
-[2] Ranftl, R., Katler, M., Koltun, V., & Kreiss, K. (2021). Vision transformers for dense prediction. In *Proceedings of the IEEE/CVF International Conference on Computer Vision* (pp. 12179-12188).
+2. Ranftl, R., Katler, M., Koltun, V., & Kreiss, K. (2021). *Vision transformers for dense prediction*. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 12179-12188).
 
-[3] Schönberger, J. L., & Frahm, J. M. (2016). Structure-from-motion revisited. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 4104-4113).
+3. Schönberger, J. L., & Frahm, J. M. (2016). *Structure-from-motion revisited*. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4104-4113).
 
-[4] Mur-Artal, R., Montiel, J. M. M., & Tardós, J. D. (2015). ORB-SLAM: Versatile and accurate monocular SLAM system. *IEEE Transactions on Robotics*, *31*(5), 1147-1163.
+4. Mur-Artal, R., Montiel, J. M. M., & Tardós, J. D. (2015). *ORB-SLAM: Versatile and accurate monocular SLAM system*. IEEE Transactions on Robotics, 31(5), 1147-1163.
 
-[5] Driess, D., Tworkowski, O., Ryabinin, M., Rezchikov, A., Sadat, S., Van Gysel, C., ... & Kolesnikov, A. (2023). PaLI-3 Vision Language Model: Open-vocabulary Image Generation and Editing. *arXiv preprint arXiv:2303.10955*. (Note: PaLiGemma is based on PaLI models, adjust reference if more specific PaLiGemma paper becomes available).
+5. Driess, D., Tworkowski, O., Ryabinin, M., Rezchikov, A., Sadat, S., Van Gysel, C., ... & Kolesnikov, A. (2023). *PaLI-3 Vision Language Model: Open-vocabulary Image Generation and Editing*. [arXiv:2303.10955](https://arxiv.org/abs/2303.10955).
 
-[6] Zhou, Q. Y., Park, J., & Koltun, V. (2018). Open3D: A modern library for 3D data processing. *arXiv preprint arXiv:1801.09847*.
+6. Zhou, Q. Y., Park, J., & Koltun, V. (2018). *Open3D: A modern library for 3D data processing*. [arXiv:1801.09847](https://arxiv.org/abs/1801.09847).
 
-[7] Plotly Technologies Inc. (2015). *Plotly Python Library*. https://plotly.com/python/
+7. Plotly Technologies Inc. (2015). *Plotly Python Library*. [https://plotly.com/python/](https://plotly.com/python/).