UV-Mamba: A DCN-Enhanced State Space Model for Urban Village Boundary Identification in High-Resolution Remote Sensing Images
Lulin Li Ben Chen Xuechao Zou Junliang Xing Pin Tao
















































UV-Mamba: A DCN-Enhanced State Space Model for Urban Village Boundary Identification in High-Resolution Remote Sensing Images
    Lulin Li^1,*, Ben Chen^1,*, Xuechao Zou^2, Junliang Xing^3, Pin Tao^1,3,† 

^1School of Computer Technology and Applications, Qinghai University, Xining, China 

^2School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China

^3Department of Computer Science  and Technology, Tsinghua University, Beijing, China 


^*Lulin Li and Ben Chen contribute equally. ^†Corresponding author. 

{lulinlee, benchen1997}@163.com, xuechaozou@bjtu.edu.cn, {jlxing, taopin}@tsinghua.edu.cn

    September 9, 2024
=========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Owing to the diverse geographical environments, intricate landscapes, and high-density settlements, the automatic identification of urban village boundaries using remote sensing images is a highly challenging task. In this paper, we propose a novel and efficient neural network model called UV-Mamba for accurate boundary detection in high-resolution remote sensing images. UV-Mamba mitigates the memory loss problem in long sequence modeling, which arises in state space model (SSM) with increasing image size, by incorporating deformable convolutions (DCN). It's architecture utilizes an encoder-decoder framework, includes an encoder with four deformable state space augmentation (DSSA) blocks for efficient multi-level semantic extraction and a decoder to integrate the extracted semantic information. We conduct experiments on Beijing and Xi’an dataset, and the results show that UV-Mamba achieves the state-of-the-art performance. Specifically, our model achieves 73.3% and 78.1% IoU on the Beijing and Xi'an datasets, respectively, representing improvements of 1.2% and 3.4% IoU over the previous best model, while also being 6 × faster in inference speed and 40 × smaller in parameter count. Source code and pre-trained models are available in the supplementary material.




Urban Village, High-resolution Remote Sensing Images, State Space Model (SSM), Segmentation




§ INTRODUCTION






Urban villages, as historical remnants in the urbanization process, present significant challenges in urban planning and management because of their low-rise and densely packed buildings, substandard environmental conditions, and outdated municipal infrastructure <cit.>. The issue of urban villages not only concerns the aesthetic and cleanliness of the city's image but also directly affects residents' quality of life, public safety, and social stability <cit.>. Traditional methods of collecting information on urban villages mainly rely on manual field surveys which is time-consuming and labor-intensive <cit.>. 

To achieve the automatic identification of urban village boundaries, the exploration of image segmentation techniques using satellite imagery has garnered widespread attention <cit.>. Several studies have employed advanced semantic segmentation models, including Fully Convolutional Networks (FCN) and U-Net, to map urban village areas <cit.>.  <cit.> utilizes adversarial learning to fine-tune the semantic segmentation network, thereby adaptively generating consistent outputs for input images across various domains. UisNet <cit.> enhances segmentation accuracy by integrating features from both remote sensing imagery and building contours through a spatial-channel feature fusion module. UV-SAM <cit.> capitalizes on the strengths of both a general model and a specialized model to apply the zero-shot capabilities of SAM <cit.> to the task of urban village boundary identification. 


However, accurately delineating the boundaries of urban villages in existing research is challenging due to two primary factors. First, the unique architectural characteristics of urban villages, including high density, narrow streets, and diverse building forms, pose inherent difficulties. Second, the limitations of CNN in capturing global information and the computational complexity of transformers <cit.>, as shown in Fig. <ref>, further complicate this task. Moreover, when ultra-high-resolution (UHR) remote sensing images are divided into smaller patches, spatial features and dependencies can be lost. 

To address the above issues, we propose the UV-Mamba model, which leverages the global modeling capability of SSM with linear complexity and the spatial geometric deformation ability of deformable convolutions. Our model mitigates the memory loss issue of SSM in long sequence modeling by employing DCN to allocate greater weights to regions of interest, thereby improving SSM's capacity to retain information across extended sequences. The main contributions of our architecture are summarized as follows:



  * We introduce UV-Mamba, a novel and efficient architecture based on SSM that effectively preserves linear computational complexity while delivering enhanced global modeling capabilities.



  * We design a DSSA module that mitigates memory loss in SSM during long-distance modeling as the sequence grows, by assigning greater weights to regions of interest using deformable convolutions.


  * We conduct extensive experiments on two cities Beijing and Xi’an in China and the results show that our method achieves superior performance, surpassing the state-of-the-art CNN-based and Transformer-based models.






§ METHODOLOGY





 §.§ Preliminaries: State Space Model


The state space model is a concept derived from the linear time-invariant systems in modern control theory. It maps a one-dimensional input signal X(t) ∈ℝ to an N-dimensional latent state h(t) ∈ℝ^N, and then projects it to a one-dimensional output signal y(t). This process can be described by the following linear ordinary differential equations (ODE):

    h'(t)    = 𝐀h(t) + 𝐁x(t), 
    
    y(t)    = 𝐂h(t),

where 𝐀∈ℝ^N × N is the state transition matrix, 𝐁∈ℝ^N and 𝐂∈ℝ^N are the projection matrix.

To better adapt to the discrete inputs in deep learning such as text sequences, 𝐀 and 𝐁 are discretized using a zero-order hold (ZOH) technique with a learnable time scale parameter Δ, which transforms the continuous SSM into a discrete SSM. The process is as follows:


    𝐀   = exp(Δ𝐀), 
    𝐁   = (Δ𝐀)^-1exp(Δ𝐀 - 𝐈) ·Δ𝐁.


After discretization, the Eq. <ref> can be represented as:

    h_k   = 𝐀h_k-1 + 𝐁x_k, 
    y_k   = 𝐂h_k,

where 𝐀 and 𝐁 represent the discretized versions of the 𝐀 and 𝐁 matrices, respectively. h_k-1 represents the previous state information and h_k represents the current state information.









 §.§ Architecture Overview


The architecture of the proposed UV-Mamba model, as depicted in Fig. <ref> (a), is composed of three principal components: 
A stem module with varying convolutional kernel sizes, a hierarchical multi-path scan encoder, and a lightweight decoder. The stem module, which performs initial feature extraction and downsamples the input image by a factor of 2, consists of four convolutional layers with 7 × 7 and 3 × 3 kernels, padding of 3 and 1, and strides of 2 and 1, respectively. The multi-path scan encoder consists of four deformable state space augmentation (DSSA) blocks, which progressively reduce the feature map size by half at each stage, resulting in feature maps of various scales relative to the model input: H/4×W/4×C_1, H/8×W/8×C_2, H/16×W/16×C_3, H/32×W/32×C_4. The decoder comprises four upsample modules, each incorporating a transposed convolution to upsample the feature map from the encoder by a factor of two, followed by two 3 × 3 convolutions for feature fusion. Finally, bilinear interpolation is used to restore the image to the input size.




 §.§ Deformable State Space Augmentation Block


For UHR remote sensing of dense urban environments, two primary challenges are refining pixel-level representation and ensuring robust global modeling for accurate boundary extraction. To address these challenges, we design the DSSA Block, which includes patch embeddings, a spatially adaptive deformable enhancer (SADE), a multi-path scan SSM module (MSSM), and patch merging, as illustrated in Fig. <ref> (b). Notably, our SADE and MSSM modules are stacked twice as intermediate modules. By assigning greater weights to regions of interest through the SADE, the issue of memory loss during global modeling with SSM can be mitigated. This approach achieves linear complexity while enhancing global modeling capabilities beyond those of SSM, enabling more effective differentiation between buildings, as shown in Fig. <ref>.


Multi-path Scan SSM Module (MSSM). 
A series of studies <cit.> have demonstrated that in SSM-based models, increasing the number of scanning directions is crucial for achieving comprehensive global modeling capabilities. To better delineate the boundaries between urban villages and adjacent communities, we aggregate scanning results from eight directions (horizontal, vertical, diagonal, and anti-diagonal, both forward and backward) to capture the complex spatial relationships of surrounding structures and to provide a thorough understanding of the contextual environment. In order to better adapt to varying input sizes, we introduce Mix-FFN <cit.>, which is more effective to provide positional information than traditional positional encoding <cit.>, by applying a 3 × 3 convolution within the feed-forward network.


Spatially Adaptive Deformable Enhancer (SADE). As shown in Fig. <ref> (c), the design of the SADE adopts a structure similar to that of the transformer <cit.>. By leveraging the spatial geometric deformation learning capabilities of the deformable convolution, it more effectively adapts to the diverse spatial distribution characteristics of urban villages. Specifically, we utilize the DCNv4 <cit.> operator for spatial feature enhancement, valued for its fast convergence and processing efficiency. The process is as follows:

    𝐲(p_0) = ∑_g=1^G∑_k=1^K𝐰_g𝐦_gk𝐱_g(p_0 + p_k + Δp_gk ),



where G denotes the total number of aggregation groups. 
For the g-th group, w_g represents the location-irrelevant projection weights, m_gkis the modulation scalar for the k-th sampling point, x_g denotes the sliced input feature map, and Δp_gk is the offset for the grid sampling location Δp_k. The extracted features are subsequently aggregated using Mix-FFN, which reduces computational complexity while maintaining the model's representational capacity. 











§ EXPERIMENTS




 §.§ Experimental Settings



Dataset. 
We use datasets from Beijing and Xi'an <cit.>, two Chinese cities with distinct architectural styles due to their significant geographical differences. Both cities feature a mix of traditional and modern buildings, creating complex urban structures that challenge our model in extracting urban village boundaries. The Beijing dataset contains 531 images, while the Xi'an dataset comprises 205 images. We divided these datasets into training, validation, and test sets in a 6:2:2 ratio. Each image has a resolution of 1024 × 1024 to ensure the inclusion of the main urban information.  

Implementation details. Our experiments are conducted on a single Tesla V100 GPU, training for 100 epochs. To prevent overfitting and improve generalization, we apply a consistent data augmentation strategy across all experiments, which included random rotation, horizontal flipping, and vertical flipping. 
The model is pre-trained on the Cityscapes dataset <cit.>, then fine-tuned on the urban village dataset. During pre-training, we utilize the Adam <cit.> optimizer with an initial learning rate of 0.001. The learning rate is warmed up for the first 10 epochs and subsequently decrease gradually to 1e-6. Cross Entropy loss <cit.> is utilized during the pre-training phase to optimize the model's performance.



The pre-trained weights are then fine-tuned on the Urban Village dataset. For fine-tuning on the Beijing and Xi'an datasets, we continue to use the Adam optimizer. The learning rate is warmed up for the first 30 epochs and then gradually decreased to 1e-6. Specifically, for the Beijing dataset, we set the learning rate to 0.0004 and use the Dice loss function <cit.>. For the Xi'an dataset, the learning rate is set to 0.0002 and the Cross Entropy loss function is employed. The models' accuracy is evaluated using Intersection over Union (IoU), accuracy (ACC) and overall accuracy (OA). The efficiency is assessed by the parameter (Params, M) and floating point operations per second (Flops, G), denoted as #P and #F, respectively, in the table for brevity.



 §.§ Ablation Studies











Image Size: To assess the impact of contextual information and spatial features on urban village boundary detection, we evaluate the model's performance using input images of varying sizes, with the results presented in Table <ref>. The results demonstrate that as image size increases, the accuracy of urban village detection consistently improves, likely due to the continuous spatial distribution of these areas. This finding highlights the importance of utilizing UHR remote sensing images for precise boundary detection.

DSSA Module: To evaluate the effectiveness of the DSSA module in UV-Mamba, we present the segmentation performance of different model variants on Beijing and Xi'an datasets in Table <ref>. The results indicate that the model's performance decreases by 2.4% and 5.5% without the SADE module. Similarly, without the MSSM module, performance drops by 2.8% and 6.7%. These findings underscore the importance of robust global modeling capabilities for accurate urban village segmentation. Furthermore, we experiment with various positional combinations of the SADE and MSSM modules within the DSSA module. The results showed that when the SADE and MSSM modules are arranged in parallel, the performance is suboptimal, achieving 72.7% and 74.9% IoU, respectively. Conversely, placing the SADE module after the MSSM module results in the worst overall model performance, suggesting that the long sequence modeling limitations of the SSM lead to feature map information loss, thereby misleading the model. In summary, these results indicate that the SADE can partially complement the global modeling capabilities of the SSM, helping to mitigate the memory loss issue when modeling high-resolution remote sensing images with the SSM.







 §.§ Comparison to the State-of-the-Arts


As illustrated in Table <ref>, UV-Mamba outperforms the advanced urban village identification models <cit.>, achieving the state-of-art performance on both the Beijing and Xi'an datasets. The visualized segmentation results are presented in Fig. <ref>. Regarding segmentation accuracy, our model demonstrates a 1%-3% improvement in IoU across the two datasets compared to the previous best urban village boundary identification  model UV-SAM with a parameter size that is 40 × smaller. Similar enhancements in performance are observed in the accuracy metrics of ACC and OA.



§ CONCLUSION


In this paper, we introduce the UV-Mamba model, which mitigates memory loss in long sequence SSM modeling, maintaining global modeling capabilities with linear complexity for precise segmentation and localization of urban village buildings in dense environments. We anticipate that our research will provide significant technical support for the modernization of urban villages, thereby advancing urban development towards increased livability, harmony, and sustainability.


IEEEtran


