# Revolutionizing Urban Safety Perception Assessments: Integrating Multimodal
Large Language Models with Street View Images

Jiaxin Zhang<EMAIL_ADDRESS>Yunqin Li  Corresponding:
<EMAIL_ADDRESS>Tomohiro Fukuda<EMAIL_ADDRESS>Bowen Wang<EMAIL_ADDRESS>
###### Abstract

Measuring urban safety perception is an important and complex task that
traditionally relies heavily on human resources. This process often involves
extensive field surveys, manual data collection, and subjective assessments,
which can be time-consuming, costly, and sometimes inconsistent. Street View
Images (SVIs), along with deep learning methods, provide a way to realize
large-scale urban safety detection. However, achieving this goal often
requires extensive human annotation to train safety ranking models, and the
architectural differences between cities hinder the transferability of these
models. Thus, a fully automated method for conducting safety evaluations is
essential. Recent advances in multimodal large language models (MLLMs) have
demonstrated powerful reasoning and analytical capabilities. Cutting-edge
models, e.g., GPT-4 have shown surprising performance in many tasks. We
employed these models for urban safety ranking on a human-annotated anchor set
and validated that the results from MLLMs align closely with human
perceptions. Additionally, we proposed a method based on the pre-trained
Contrastive Language-Image Pre-training (CLIP) feature and K-Nearest Neighbors
(K-NN) retrieval to quickly assess the safety index of the entire city.
Experimental results show that our method outperforms existing training needed
deep learning approaches, achieving efficient and accurate urban safety
evaluations. The proposed automation for urban safety perception assessment is
a valuable tool for city planners, policymakers, and researchers aiming to
improve urban environments.

###### keywords:

Urban Safety Perception , Large Language Models , Retrieval , Deep Learning ,
Street View Images

[label1]organization=Architecture and design college, Nanchang
University,addressline=No. 999, Xuefu Avenue, Honggutan New District,
city=Nanchang, postcode=330031, country=China, [label2]organization=Division
of Sustainable Energy and Environmental Engineering, Osaka
University,addressline=2-1,Yamadaoka, city=Osaka, postcode=5650871,
state=Suita, country=Japan, [label3]organization=Institute of Datability
Science, Osaka University,addressline=2-1,Yamadaoka, city=Osaka,
postcode=5650871, state=Suita, country=Japan,

##  1 Introduction

Perceptions of safety in residential environments positively impact citizens’
overall happiness [1]. Urban safety perception emphasizes individuals’
subjective feelings and psychological responses to the built environment in
specific contexts [2]. Early researchers collected safety perception data
through field surveys, on-site interviews, and questionnaires to evaluate the
safety level of a given area [3]. However, these labor-intensive data
collection methods are limited in scope, time-consuming, and costly.

In recent years, studies utilizing computer vision technology to uncover
hidden city profiles through Street View Images (SVIs) have emerged to assess
city-scale safety perceptions [4]. A primary challenge is the efficient
acquisition of quantifiable data for training deep learning models to achieve
robust model performance, given the subjective nature of safety perception
[5]. The development of MIT’s Place Pulse project marked a significant shift
in urban perception data collection methods [6]. By employing an online
crowdsourcing strategy, participants compared pairs of images across six
perceptual dimensions, resulting in the creation of the large-scale urban
perception dataset Place Pulse 2.0 [7]. Despite the demonstrated efficiency
and effectiveness of deep learning models trained on Place Pulse 2.0 for
predicting urban-scale safety perception scores [8], the compilation of this
dataset demanded considerable human effort, resources, and financial
investment. Therefore, improving the efficiency of annotating subjective
perception datasets and the transferability of models without increasing costs
remains a key area for future efforts.

Recently, large language models (LLMs) [9, 10] have demonstrated their
proficiency in numerous sophisticated language understanding and generation
tasks, significantly transforming the field of natural language processing.
These models, which are trained on extensive text corpora, are adept at
capturing the subtleties of human language, ranging from article writing and
summarization to producing coherent and contextually appropriate text. This
capability has been expanded to multimodal large language models (MLLMs) [11],
which have achieved remarkable successes. Models like GPT-4V [12] are now
capable of not only understanding and generating text but also interpreting
and analyzing visual information. This convergence of visual and linguistic
processing allows them to respond to queries about image content, effectively
narrowing the divide between visual perception and linguistic expression [13,
14, 15]. By processing and comprehending complex visual scenes, these models
can identify objects, detect patterns, and even deduce emotions or actions
depicted in images [16, 17, 14]. This raises a question: Can MLLMs be used to
replace humans in obtaining safety perceptual scoring?

This research investigates the possibility of using MLLMs to automatically
obtain safety scores from images. The study began by assembling an anchor set
of 1,000 SVIs from Chengdu, China. Chengdu was chosen due to its diverse urban
environment and rapid infrastructure growth, making it an ideal case for
examining urban safety perceptions using MLLMs and SVIs. Each image in the
anchor set was annotated with a safety score derived from comprehensive human
assessments. These scores serve as a benchmark for evaluating the efficacy of
the proposed approach. These scores serve as a benchmark for evaluating the
efficacy of our approach. MLLMs and predefined prompts are utilized to
facilitate automatic scoring. Our goal is to use MLLMs to replace the safety
comparison tasks traditionally performed by humans, mimicking the data-making
process of the Place Pulse 2.0 dataset. Through this automated process, the
model assigned a safety score to each SVI. A statistical analysis using the
coefficient of determination ($R^{2}$) revealed a correlation of 0.3263
between the model-generated scores and human assessments. This finding
suggests that MLLMs, particularly state-of-the-art models like GPT-4V [12],
can effectively undertake automatic scoring tasks.

Furthermore, a novel urban-wide safety scoring system that leverages a
K-Nearest Neighbors (K-NN) retrieval method based on Contrastive Language-
Image Pre-training (CLIP) [18] features is introduced. This innovative
approach does not require a training phase, instead utilizing a weighted
aggregation technique to enhance performance. Preliminary experimental results
demonstrate that this method significantly outperforms traditional models that
require extensive training, suggesting a promising direction for scalable,
real-time urban safety assessments. Our findings not only underscore the
capabilities of MLLMs in interpreting complex urban data but also pave the way
for further advancements in automated SVIs-based safety analysis.

Our contributions to this study can be distilled into three primary aspects:
(1) A safety dataset is constructed to serve as a benchmark for urban
perception, which is based on human evaluations of SVIs from Chengdu. (2) We
introduced MLLMs to analyze urban safety perception and compared the results
with human perception outcomes, revealing no significant differences between
the two. (3) An urban-wise scoring methodology utilizing CLIP features and
K-NN retrieval is proposed, outperforming previous approaches that requires
training.

Figure 1: Comparison of manually scored data and multimodal large language
model scored data for urban safety perception. (a) Manually comparing pairs of
SVIs for safety perception to create a training set, and (b) inputting two
SVIs simultaneously into the MLLM to determine which image is perceived as
safer.

##  2 Related Works

###  2.1 Measuring Urban Safety Perception via SVIs

Despite numerous efforts in urban safety research [1, 19, 20, 21], a clear and
unified definition of perceived safety is lacking. Different studies define
perceived safety in various ways. Mehta et al. define perceived safety as an
individual’s sense of security influenced by social and physical factors [22].
Mouratidis et al. describe perceived safety as the level of comfort and risk
perceived in the environment[23]. Similarly, Qiu et al. emphasize that
perceived safety is not an objective quantification but rather a subjective
perception of the place [2].

Some scholars equate perceived safety with fear of crime, but this is
conceptually inaccurate [24]. In these studies, perceived safety is often used
as an indicator to quantify urban crime rates [25]. Zhang et al. show a
mismatch between perceived safety and actual crime rates, and potential
contradictions between urban spatial characteristics and “perception bias” in
crime [26]. Kang et al. propose that the fear of crime is a trait reflecting
individual differences in the experience of fear, whereas perceived safety is
a situational and instantaneous state [27]. In essence, perceived safety is a
transient feeling influenced by the immediate perception of danger or threat,
distinguishing it from the more enduring and pervasive nature of the fear of
crime, which spans across various contexts and timeframes.

Field surveys, interviews, and questionnaires are the main methods for
collecting perceived safety data [3]. For instance, since 1973, the U.S.
National Crime Survey (NCS) has collected data by asking: How safe do you
feel, or would you feel safe walking alone in your neighborhood at night?
While these methods provide detailed insights into human perceptions and
social background differences, they are limited by research scope, time-
consuming data collection, and high costs [8]. When quantifying perceived
safety, it is crucial to balance accuracy and scalability, ensuring subtle
differences are captured while accommodating broader applications [28].

With the proliferation of SVIs and the widespread application of machine
learning, crowdsourcing surveys using SVIs combine accuracy with scalability
[4]. These images offer broad availability, ample sample sizes, and consistent
spatial granularity, comprehensively representing the visual and morphological
features of urban environments [29]. These methods not only provide the urban
appearance in specific scenarios but also give volunteers instantaneous visual
impressions, enabling the quantification of perception assessments [28]. To
better understand the role of perceived safety in urban environments, it is
essential to consider quantification methods on a global scale. While manual
statistical methods can capture individual and area-specific nuances, they are
challenging to extend to systematic global studies. Therefore, combining prior
knowledge of human safety perception with the capabilities of multimodal large
language models may enable machine-supervised understanding, leading to
advancements in broader applications and more precise quantitative analysis.

###  2.2 Quantifying Urban Perception At a Large Scale

Urban planners and sociologists have identified correlations between unsafe
visual characteristics and issues such as crime and lower educational outcomes
[30, 31]. Nasar et al. [32] posit that the visual and physical features of a
city profoundly influence residents’ cognition and emotional responses to
urban spaces. However, quantitatively assessing perceptions of urban visual
environments has been challenging due to limitations in data collection
methods, such as insufficient sample sizes and reliance on interviews and
surveys[33]. With advancements in image capture technology, SVIs have
gradually become a significant medium for reflecting urban appearance.

The study by Griew et al. evaluated the quality of SVIs using expert
assessments or crowdsourced ratings, where participants scored the images on a
scale from 1 to 10 [34]. Salesses et al. created the Place Pulse 1.0 dataset
by comparing pairs of SVIs to gather perceptions of urban environments. In
this dataset, participants evaluated randomly selected image pairs, answering
questions like Which place looks safer? MIT’s Place Pulse project marked a
shift in urban perception data collection methods [7]. Based on this dataset,
Naik et al. [35] proposed the Streetscore algorithm, using support vector
regression (SVR) and image features to predict perceived street safety. This
algorithm analyzes features like texture, color, and shape to predict
Streetscore, thereby automatically generating an urban appearance dataset for
21 U.S. cities [36]. However, because Streetscore was primarily trained on
images from New York and Boston, its accuracy in measuring urban perception
globally is limited.

Advances in deep learning for image recognition have driven the widespread
application of convolutional neural network (CNN) models in urban visual
quality perception studies based on the Place Pulse 1.0 dataset [37, 38, 39].
Dubey et al. [40] combined online crowdsourcing and deep learning techniques
to create the Place Pulse 2.0 dataset, which includes 110,988 images and 1.17
million pairwise comparisons from 56 cities worldwide, rated by 81,630
volunteers. This dataset covers six perceptual dimensions (Safe, Lively,
Beautiful, Wealthy, Depressing, and Boring). They also designed a Ranking
Streetscore-CNN model to predict visual attributes of SVIs. These studies have
not only improved data collection efficiency but also expanded the
geographical coverage of samples, laying the foundation for global urban
perception research.

Currently, the feasibility of machines replacing humans in rating SVIs is
becoming a research focus. Naik et al. [41] used SVIs and the Place Pulse 2.0
dataset to study changes and drivers in visual quality perception of community
appearances. However, the Place Pulse 2.0 dataset still has limitations in
transferability, as deep learning models trained on this dataset perform
poorly in predicting perceptions of SVIs from developing regions [5].
Additionally, measuring models for street visual quality perception can be
influenced by factors such as the age, gender, occupation, behavior, and the
time of evaluation of the subjects [28, 4, 42, 43, 44, 45, 46, 47].

Figure 2: The overall pipeline of our proposed method attaches an anchor set
with safety perception scores via automatic ranking, followed by an urban
scoring method to analyze the whole city.

###  2.3 Multimodal Large Language Models in Human Perception

In recent years, the emergence of LLMs has brought about revolutionary changes
in the field of Natural Language Processing (NLP), with models such as ChatGPT
[48], GPT-4 [49], PaLM [50], and LLaMa [51] leading the charge. LLMs, owing to
their vast corpora and intensive training computations, have demonstrated
impressive capabilities in zero-shot and few-shot tasks, as well as in more
complex tasks like mathematical problem-solving and common-sense reasoning.
For instance, the advent of ChatGPT has highlighted the potential of LLMs in
understanding human intent, reasoning, and following instructions to generate
the required responses for specific tasks. Meanwhile, the introduction of
GPT-4 [12] has unlocked tremendous potential for multimodal perception, which
is crucial for real-world foundational capabilities.

Creating models for street view visual quality perception faces challenges of
being time-consuming and potentially biased compared to human perception [27].
The emergence of MLLMs offers new opportunities for developing universal
models. Y. Zhang et al. [52] proposed a purely visual approach that can
generate textual descriptions of SVIs. Nevertheless, research on obtaining
street visual quality perception evaluations through MLLMs via pairwise
comparisons of SVIs remains absent. Rating methods may affect data consistency
due to respondent habits and differences in understanding scales, while
pairwise comparison methods are intuitive and reduce cognitive burden,
suitable for decision-making scenarios with multiple options [53, 54]. MLLMs
combine language understanding and image feature learning, offering the
potential for comprehensive understanding and prediction of street view
perceptions. This study aims to investigate whether MLLMs, pre-embedded with
knowledge of safety perceptions in built environments, can emulate human
evaluations of SVI safety. By doing so, it seeks to address the time-consuming
nature of pairwise comparisons in image perception tasks and the
transferability issues of pre-trained models.

##  3 Methods

###  3.1 Overview of Our Method

In this study, a novel approach was utilized to score the perceived visual
safety of SVIs by leveraging a multimodal large language model (MLLM). Figure
1 shows the workflow of manually scored data and MLLM-scored data for urban
safety perception. In previous methods, SVIs were manually scored by human
annotators who compared pairs of images to determine which appeared safer.
These manual scores formed the training dataset for the deep learning model.
In our proposed method, the MLLM was employed to automatically score a new set
of SVIs. The MLLM was pre-trained with prior knowledge from existing visual
safety research, enabling it to determine which of the two SVIs is safer based
on established criteria. This allows for the statistical acquisition of
relatively objective SVI safety perception scores through extensive
comparisons of SVIs. This automated scoring by the MLLM efficiently increased
the scale of dataset that can be annotated, demonstrating the model’s
potential to replicate and scale the human perception of street safety.

The safety analysis process is illustrated in Figure 2, and we took Chengdu’s
central area as an example. First, SVIs were collected via Baidu Maps, and a
representative anchor set was selected. Then, a MLLM was used to automatically
assign safety scores to these images. To assess the overall safety of the
city, a pre-trained CLIP model was employed to extract features from both the
anchor set images and representative SVIs of the city. Subsequently, a K-NN
retrieval method, combined with a weighted aggregation technique, was used to
compute the final scores. Finally, these scores were projected onto the city
map to provide a comprehensive visual representation of urban safety.

###  3.2 Dataset Making and Anchor Set

We used Baidu Maps to collect 69,681 SVIs points of Chengdu (each point have
four directions 0°, 90°, 180°, and 270°), denoted as
$\mathcal{X}=\\{x_{j}^{l}\\}$, where $j$ is the index of SVI and $l$ is the
index of direction. From all the SVIs in Chengdu, we randomly sampled 1,000
representative images to form our anchor set $\mathcal{A}$. We then created a
safety perception dataset of SVIs following the methodology used in the Place
Pulse 2.0 dataset.

Anchor Set We employed an online crowdsourcing method and invited 50
volunteers to randomly compare 1,000 representative SVIs in Chengdu to form
our anchor set $\mathcal{A}$. The ages of the volunteers ranged from 18 to 65
years (mean age = 35), ensuring diversity in age, gender, and occupational
backgrounds: 20 % were aged 18-25, 30 % aged 26-35, 25 % aged 36-45, 15 % aged
46-55, and 10 % aged 56-65. The gender distribution was balanced, with 52
percent male and 48 % female participants. Occupational backgrounds included
students (36 %), white-collar workers (10 %), freelancers (14 %), engineers
(24 %), educators (10 %), and others (6 %). Each image is compared to 40 other
images in average.

Prior to the voting process, participants were provided with basic safety
guidelines. As shown in Table 1, these guidelines are derived from seminal
works on street safety perception by Gehl [19], Ashihara [20], and Jacobs
[21], among others, as well as recent theoretical advancements in the field.
To systematically collect safety perception data, we developed a program that
utilizes a pairwise comparison method to assess images, ensuring that each
pair of images receives votes from at least 40 participants. In this program,
participants are presented with a randomly selected pair of images and are
asked to choose the image they perceive as safer based on a predetermined
question Which place looks safer or to mark Cannot decide if they are unable
to make a judgment, as shown in Figure 1(a). The pairwise comparison method,
compared to subjective human image rating methods [7], effectively reduces
subjective bias and fully utilizes the visual information of the images.
Throughout the dataset creation process, participants were not informed of the
specific spatial locations of the images.

Table 1: Basic safety criteria $C$ provided to participants for ranking tasks.

Type | Descriptions
---|---
Safe | Areas with high pedestrian activity, such as commercial buildings or residential zones
| Public service facilities, including police stations and hospitals
| Well-maintained and organized street trees
| Sidewalks that are clean and in good repair
| Active and clean downtown roads
| Clearly marked and easily accessible public transport stops
| Clear and visible road signs and directions
| Well-maintained street decorations or public spaces
| Presence of well-maintained greenery and parks
Dangerous | Buildings that are damaged or abandoned
| Walls that are blocked or in disrepair
| Remote rural or suburban roads with little traffic
| Areas where garbage is piled up or the environment is neglected
| Active construction sites with insufficient safety measures
| Areas lacking sufficient traffic lights
| Complex and confusing traffic systems
| High traffic areas with disorganized vehicle and pedestrian flow
| Open land that is desolate and uninhabited
| Narrow, enclosed spaces
| Long and narrow roads or tunnels with poor visibility

###  3.3 Automatic Ranking

As discussed earlier, the anchor set $\mathcal{A}$ functions as a prototype
for scoring. The safety ranking within this set can represent the score
distribution of SVIs across the entire city. The critical challenge here is
determining how to assign a specific score to each image. One possible method,
as outlined in Section 3.2, is through human voting. However, this approach is
time-consuming, particularly when dealing with a large number of images.
Additionally, human judgment is inherently subjective, necessitating a
substantial number of participants to mitigate bias and achieve more objective
results.

We thus proposed to use recent advanced MLLM to realize the scoring task
automatically. Given two SVIs $x_{1}$ and $x_{2}$, the safety comparison is
responded by:

$Response=MLLM(Q,C,x_{1},x_{2}),$ (1)

where $Q$ represents the question formulated using a predefined prompt, $C$ is
the criteria shown in Table 1. In this prompt structure, we posit that the
MLLM acts as an urban environment specialist, tasked with comparing the
inputted SVIs and providing a brief rationale. Additionally, we supply the
criteria $c$ employed in human scoring as a reference to guide the model’s
assessment. We show one sample of comparison using GPT-4V as MLLM in Table 2.

Table 2: A demonstration of comparing safety between two SVIs utilizing an
MLLM, with GPT-4V as the current example. We begin by inputting two SVIs.
Next, based on a predefined prompt, we engage the MLLM to assess and explain
the relative safety of the two images, subsequently determining the safer
option.

Visual Input Example for Safety Comparison
---
|
Question |  You are an urban environment expert. Here is the definition of safe and dangerous for city scenes: {$C$}. Now help me to compare the two input images and tell me which one is safer. Give me a choice from A: First Image or B: Second Image. C: Unable to compare. You also need to briefly explain your choice.
Response |  Choice: A: First Image. The first image depicts a well-maintained urban space with clear pedestrian pathways, neat rows of trees, and a vibrant garden area, which typically indicates a well-cared-for and safer environment. There are traffic signals in place and the presence of flowering plants suggests regular maintenance. Additionally, the area is open and visible, which can contribute to a sense of safety. In contrast, the second image shows a neglected area with debris, a lack of pedestrian infrastructure, and construction that appears unmanaged. The environment is less inviting, which may lead to a lower perceived level of safety. The presence of construction indicates potential hazards, and the overall disarray could be indicative of neglect. These elements typically correlate with a higher-risk environment and can contribute to a sense of unsafety.

Our automated ranking system is conceived to replicate the process of human
voting. Each image $a_{i}\in\mathcal{A}$ is compared with $N=40$ images drawn
from a randomly chosen subset $\mathcal{R}^{i}$ of $\mathcal{A}$ 111Note that
$a_{i}$ itself is not included in $\mathcal{R}^{i}$. Furthermore, we maintain
an index collection $\mathcal{S}$ for each $a_{i}$ to log the cumulative
instances it is deemed safe, assigning 1 point per such designation. Hence,
the ranking is denoted by the following equation:

$\mathcal{S}=\oint_{i}^{I}\oint_{n}^{N}\Theta(MLLM(Q,C,a_{i},r^{i}_{n})),$ (2)

where $\oint$ is the loop operation, $I$ symbolizes the total count of images
in $\mathcal{A}$, $r^{n}_{i}$ is one image from $\mathcal{R}^{i}$ and $\Theta$
represents the tallying function that records an image as safe as determined
by the MLLM. Note that if the response is “Unable to compare” the result will
not be counted.

After the ranking process, we obtain $\mathcal{S}$ that encapsulates the
safety score $s_{i}$ for each $a_{i}$. To facilitate comparison and
interpretation, it is imperative to normalize these scores to a uniform range.
We accomplish this by mapping the scores to a scale from 0 to 10, which
enables a standardized evaluation of safety across different images. The
normalization formula is given by:

$\mathcal{S}^{*}=10\frac{\mathcal{S}-min(\mathcal{S})}{max(\mathcal{S})-min(\mathcal{S})},$
(3)

where $max$ and $min$ denote the operations to extract the highest and lowest
scores within the set $\mathcal{S}$, respectively. This standardization
process not only renders the scores more intuitive by placing them on a decile
scale but also mitigates the effects of outliers, thereby providing a more
accurate and robust comparison of safety levels.

###  3.4 Urban Scoring Via CLIP and K-NN Retrieval

Given the extensive number of SVIs contained within $\mathcal{X}$, directly
appraising the safety of an entire city via the ranking method delineated in
the previous section is an infeasible task. A common approach to overcome this
challenge would be to develop a dedicated scoring deep learning model [7] that
processes individual SVIs and outputs corresponding safety scores based on the
anchor set. Nonetheless, the development of such a model is fraught with
difficulties, compounded by the potential for inaccuracies, particularly when
the available data is sparse.

To address these problems, we propose an innovative solution that leverages a
pre-trained CLIP model in conjunction with a K-NN retrieval system. This
approach harnesses the robust feature extraction capabilities of CLIP to
analyze the visual content of SVIs and understand their contextual nuances in
relation to safety. By integrating these features with a K-NN retrieval
mechanism, we can effectively match each city’s SVI with its closest
counterparts in the anchor set. This process not only bypasses the intensive
data requirements of training a bespoke model from scratch but also enhances
the accuracy of the safety scores through the use of a sophisticated, pre-
established knowledge base. Thus, we can extrapolate a comprehensive safety
score for the city, offering a nuanced and data-rich safety landscape.

Here, we represent the pre-trained CLIP model as $M$. The feature extraction
process for all SVIs within the set $\mathcal{A}$ is mathematically expressed
as follows:

$\mathcal{F}=\\{M(a_{i})\mid a_{i}\in\mathcal{A}\\},$ (4)

where $\mathcal{F}$ is the collection of feature vectors, and each $f_{i}$ is
a vector extracted from the last layer of the CLIP image encoder.

For any given SVI $x^{l}_{j}$ from $\mathcal{X}$, we denote its extracted
feature vector by $q^{l}_{j}$. The similarity measurement between $q^{l}_{j}$
and each anchor feature $f_{i}$ is computed using the cosine similarity metric
$sim$, resulting in the feature distance:

$\mathcal{D}^{l}_{j}=\\{sim(q^{l}_{j},f_{i})\mid f_{i}\in\mathcal{F}\\},$ (5)

where $\mathcal{D}^{l}_{j}$ encapsulates the similarity between $x^{l}_{j}$
and each SVI in the anchor set.

To derive a weighted relevance score for $x^{l}_{j}$, we aggregate the scores
of its top K-nearest neighbors. This aggregation is performed by weighting the
score of each neighbor by its relative distance, which emphasizes closer
neighbors and potentially increases the reliability of the relevance score:

$o^{l}_{j}=\sum_{k=1}^{K}s_{k}\left(\frac{d_{k}}{\sum_{k=1}^{K}d_{k}}\right),$
(6)

where $s_{k}$ is the recorded score of the k-th neighbor in the ranking list
$\mathcal{S}^{*}$ of anchor set (defined in Equation 3), and $d_{k}$
represents the distance between $x^{l}_{j}$ and its k-th nearest neighbor. The
normalization by the sum of distances ensures that the weighted scores sum to
one, maintaining a probabilistic interpretation of the relevance scores. This
methodological framework allows for nuanced insights into the relationships
between images based on deep feature similarities, providing a robust result
for image-based retrieval scoring.

The final safety score for data point $j$ is calculated by averaging the
scores from four SVIs associated with it, as follows:

$o_{j}=\frac{1}{4}\sum_{l}^{4}o^{l}_{j}$ (7)

After the calculation for all the data points, we will draw the overall safety
map.

##  4 Results

###  4.1 Experimental Settings

In selecting the MLLMs, we have incorporated four recent state-of-the-art
(SOTA) open-source methods: LLaVA-NeXT [55], Phi3-V [56], LlaMA-Adapter [57],
InstructBLIP [58], and MiniGPT-4 [17]. Additionally, we evaluate our approach
using a closed-source MLLM, GPT-4V, accessible through the API (gpt-4-vision-
preview). The temperature value is set as 0.05 for all the MLLMs. For the pre-
trained CLIP model, we employ a version equipped with a ViT-B/16 backbone,
which outputs a 512-dimensional feature vector. 80% of the data in the anchor
set are used for training and the rest for evaluation. The K-NN retrieval is
set with a default maximum $K$ value of 10. All experiments are conducted on a
GPU server equipped with four Nvidia A40 GPUs.

Following [7], we adopted the coefficient of determination $R^{2}$ to evaluate
the difference between the two results, which is formulated as follows:

$\displaystyle SST=\sum^{T}_{t=0}(y_{t}-\bar{y})^{2},$ (8) $\displaystyle
SSE=\sum^{T}_{t=0}(y_{t}-\hat{y}_{t})^{2},$ (9) $\displaystyle
R^{2}=1-\frac{SSE}{SST},$ (10)

where $y_{t}$ are the observed values (from the human vote), $\bar{y}$ is the
mean of the dependent variable, and $\hat{y}_{t}$ are the values predicted by
the model (MLLMs or CLIP based retrieval). We also adopt Mean Absolute Error
(MAE) for further evaluation.

Table 3: The score difference between human vote and automatic ranking via
MLLM. The mean is calculated by the Mean Absolute Error (MAE) of the score
difference for each SVI in the anchor set. Std is the standard deviation. Max
and min are the maximum and minimum difference values, respectively. R2 is the
coefficient of determination.

Methods | Metrics
---|---
Mean ± Std $\downarrow$ | Max $\downarrow$ | Min $\downarrow$ | R2 $\uparrow$
GPT-4V [49] | 0.9157 ± 0.6916 | 3.7605 | 0.0027 | 0.3263
LLaVA-NeXT [55] | 1.0514 ± 0.7332 | 3.9662 | 0.0035 | 0.2637
Phi3-V [56] | 1.1902 ± 0.8099 | 4.1321 | 0.0032 | 0.2003
LlaMA-Adapter [57] | 1.2377 ± 0.8386 | 4.6744 | 0.0031 | 0.1817
InstructBLIP [58] | 1.3328 ± 0.9031 | 4.8947 | 0.0039 | 0.1452
MiniGPT-4 [17] | 1.3501 ± 0.9178 | 5.0100 | 0.0046 | 0.1319

###  4.2 Evaluation of Automatic Ranking

The primary objective of this research is to investigate the capability of
MLLMs to align with human judgments in the task of urban safety ranking. This
alignment is crucial for developing reliable AI tools that can assist in urban
planning and public safety assessments. As illustrated in Table 3, we conduct
a comparative analysis of the performance of several leading MLLMs.

It is evident that GPT4-V excels in several key performance metrics,
outperforming the other models. Specifically, GPT4-V has the lowest Mean
Absolute Error (MAE) (Mean ± Std) at 0.9157 ± 0.6916, indicating the smallest
discrepancy between its scores and human votes. Its results are also more
stable, and less affected by extreme values. The maximum error (Max) for
GPT4-V is 3.7605, the lowest among all models, showing it maintains a smaller
error range across all instances. Although its minimum error (Min) of 0.0027
is similar to other models, its overall performance is superior when
considering other metrics. Additionally, GPT4-V’s coefficient of determination
R2 is 0.3263 222Considering the subjectivity inherent in human evaluations, we
believe that the performed R² score is sufficient to demonstrate that GPT4-V
can align with human judgment in the task of urban safety ranking.,
significantly higher than the other models, indicating its stronger ability to
explain data variability and better fit. In contrast, the other models exhibit
comparatively inferior performance across these metrics.

Figure 3: Probability density of scores for the anchor set calculated from
human votes and MLLM automatic rankings (LLaVA-NeXT and GPT-4V), with the
x-axis representing scores from 0 to 10 and the y-axis showing the
distribution density.

Figure 3 shows the probability density of scores for the anchor set,
calculated from human votes and MLLM automatic rankings (including LLava-NeXT
and GPT4-V). The x-axis represents the score range from 0 to 10, and the
y-axis shows the distribution density. From the figure, it can be observed
that the probability density distribution curve for human scores, represented
by the solid red line, is relatively smooth, with a high density of around 4.
This indicates that most human scores are concentrated in this range. The
probability density distribution for GPT4-V, indicated by the blue dotted
line, matches the distribution of human scores, peaking around a score of 4
(although some density difference indicates that GPT-4V is more neural). This
suggests that the GPT4-V model is able to simulate human scoring distribution
quite well. However, the probability density distribution for LLava-NeXT,
shown by the yellow dashed line, differs somewhat from the human score
distribution. The distribution for LLava-NeXT is deviated to the right,
indicating that LLava-NeXT’s scores do not completely align with human scores
in this range.

###  4.3 Evaluation for Scoring

Our analysis reveals that CLIP serves as a highly effective backbone for
extracting image features. Rather than training a separate classifier, we
leverage these features directly to calculate the similarity between SVIs.
This approach not only simplifies the process but also enhances the efficiency
and accuracy of scoring measurements.

Figure 4: UMAP visualization of the anchor set’s image features extracted by
CLIP, with safety scores represented by a color gradient from red (dangerous)
to green (safe) and accompanied by sample images for reference. Figure 5:
Depicted here are examples of various types of SVIs, categorized based on
their assigned risk scores. From left to right, the classifications are as
follows: Dangerous, indicated by a score less than 3; Neutral, characterized
by scores ranging from 3 to 6; and Safe, denoted by scores exceeding 6. Each
category reflects the relative safety level, with Dangerous representing high-
risk conditions, Neutral indicating moderate risk, and Safe signifying low or
negligible risk.

As shown in Figure 4, as proof, we showcase the results of uniform manifold
approximation and projection (UMAP) [59] visualization of image features
extracted using CLIP. Each point represents an image, with its safety score
indicated by a color gradient ranging from red to green, where red denotes
danger and green indicates safety. The distinct clusters formed by the points
of different colors demonstrate that CLIP effectively captures image features,
grouping similar image features together. The clear distribution of safety
scores, as shown by the color gradient, indicates that the features extracted
by CLIP can accurately reflect differences in image safety. UMAP brings
together image points with similar features, suggesting that the features
extracted by CLIP have a good distinguishing capability and can effectively
reflect the similarity between images. This supports the choice of directly
using CLIP features for similarity calculation, rather than training a
classifier, making the method both efficient and accurate for scenarios
requiring quick assessment of image similarity. Some samples are shown in
Figure 5.

Additionally, in Table 4, we present the evaluation results of different
methods for safety score prediction. Our method is marked with an asterisk
(*), and all other methods are fine-tuned with a regression head to predict
the score. The R2 values are used to assess the performance of each method.
ResNet-50 and ViT-B/16 (ImageNet pre-training) have an R2 value of 0.1363 and
0.1077, indicating their relatively weak prediction capability for safety
scores. CLIP pre-training (RN50 and VIT-B) achieves an R2 value of 0.3552 and
0.3917, significantly higher than ImageNet pre-training, demonstrating better
performance in extracting image features and predicting safety scores. Our
CLIP+KNN method achieves an R2 value of 0.4392, the highest among all methods.
This method directly utilizes the features extracted by CLIP and combines them
with KNN for score prediction without the need for an additional regression
head. This result further proves the effectiveness and advantage of our
approach.

Table 4: Evaluation on different methods for safety score prediction. Our
method is attached with *, and all other methods are finetuned with a
regression head to predict the score.

| ResNet-50 | ViT-B/16 | CLIP (RN50) | CLIP (VIT-B) | CLIP+KNN∗
---|---|---|---|---|---
R2 | 0.1363 | 0.1077 | 0.3552 | 0.3917 | 0.4392

Figure 6: Analysis of using top K neighbors for our K-NN based retrieval,
comparing various backbone architectures with different pre-training patterns,
including CLIP and ImageNet.

We also provided an ablation study for the choice of $K$. Figure 6 illustrates
the R² performance of various backbone architectures using different numbers
of neighbors ($K$ values) for K-NN based retrieval. It compares the
performance of architectures pre-trained with CLIP (CLIP-ViT-B and CLIP-RN-50)
and ImageNet (ImageNet-ViT-B and ImageNet-RN-50). Among them, CLIP-ViT-B
consistently performs the best across all $K$ values, with its R2 value
steadily increasing as $K$ increases from 1 to 10, peaking at 0.4392, and then
slightly decreasing but remaining high. CLIP-RN-50 also shows improvement with
increasing $K$ values. In contrast, both ImageNet-ViT-B and ImageNet-RN-50
perform significantly worse across all $K$ values, with lower R2 values
indicating weaker feature extraction and similarity calculation capabilities.
Overall, the result demonstrates that the general trend of increasing R2
values with larger $K$ values suggests that using more neighbors improves
prediction accuracy, although the benefits level off or slightly decreases
beyond a certain point.

Figure 7: Distribution of urban streets with different types of safety and
representative cases.

###  4.4 Safety Perception Clustering at City Scale

The results in Figure 7 indicate that urban safety perception in the central
urban area of Chengdu can be categorized into four types: high-safety, medium-
high-safety, medium-low-safety, and low-safety, accounting for 19.4%, 33.3%,
30.0%, and 17.1%, respectively. Using hierarchical clustering based on the
local effects of SVI visual indicator variables, we can explore the impact of
different physical elements’ visual proportions on urban safety perception.
Figure X shows the spatial distribution of these clusters and the density
plots of their main visual indicator variables.

The high-safety cluster is primarily located in the city center and areas with
well-developed built environments, characterized by a high Visual Enclosure
Index (VEI) and Visual Mobility Index (VMI) but a low Green View Index (GVI).
Dense buildings and heavy traffic increase human activity and surveillance,
thereby providing a higher perception of safety. The medium-high-safety
cluster is mainly distributed in the urban core and around major traffic
routes. Although its VEI, VMI, and Sky Openness Index (SOI) are slightly lower
than those of the high-safety cluster, it has a higher Visual Walkability
Index (VWI) and GVI. A good walking environment and moderate greening enhance
residents’ outdoor activity time and social interaction, thereby improving
safety perception.

The medium-low-safety cluster is located on the periphery of the urban center,
with moderate levels of VMI, VWI, GVI, and SOI. The scattered layout of
buildings and roads, along with poor greening and walking conditions, may lead
to a weaker safety perception in these areas. The low-safety cluster is mainly
distributed in the outermost areas of the city, near the fourth ring road.
This cluster features high SOI and GVI but low VEI. Low building density and
reduced human activity, coupled with a lack of necessary surveillance and
public activities, result in a decrease in safety perception.

###  4.5 Nonlinear Associations of Factor Identification

Figure 8: Global and local relative importance diagram of SHAP model.

By quantifying the contribution of each input variable to the prediction
outcomes through SHapley Additive exPlanations (SHAP) values, we can gain a
deeper understanding of whether the results of the prediction model, based on
MLLM scoring, are consistent with human experiences of street safety. We
utilized the eXtreme Gradient Boosting (XGBoost) model to establish the
relationship between safety perception scores and 19 physical elements of the
SVI, and employed SHAP values to interpret the model results. We utilized
DeepLab v3 with a pre-trained model on the Cityscape dataset to perform
semantic segmentation of 19 physical elements in the SVIs and calculated the
proportion of each element in the images.

Figure 8 illustrates the global importance and local explanation plots of
various physical element variables. The variables are ranked in descending
order of global importance, showcasing the top nine indicators. In the local
explanation plots, red to blue dots represent feature values from high to low,
respectively, with the x-axis SHAP values indicating the positive and negative
impacts on safety perception. Overall, the three most significant contributing
factors are buildings, roads, and cars, all of which positively influence
safety perception. Partial dependence plots further visualize the SHAP values
for each sample point, with the x-axis representing the normalized variable
size and the y-axis representing SHAP values. We presented the top nine
independent variables in terms of relative importance (Figure 9). These plots
help explore the nonlinear and threshold effects of individual independent
variables on the dependent variable.

Figure 9: Local dependence diagram of urban multi-source data on safety
perception.

The results indicate that the proportion of different elements has significant
nonlinear effects on safety perception. The proportions of building elements,
roads, cars, and sky elements exhibit complex influences on safety perception.
Lower proportions of building elements negatively impact safety perception,
but a moderate increase in building density can enhance it. However, when the
proportions of building, road, and car elements exceed a certain threshold,
their positive impact diminishes and may even negatively affect the perception
of safety. Similarly, while the increase in the proportion of sky elements is
positively correlated with enhanced safety perception, at higher proportions,
this effect stabilizes. Excessive sky view may imply a lack of shelter and
privacy, thereby reducing the perception of safety.

In contrast, the proportions of wall and truck elements are significantly
negatively correlated with safety perception. While wall elements can provide
boundaries and a sense of enclosure, enhancing area safety to some extent,
excessive walls can lead to oppressive spaces and visual monotony, reducing
the attractiveness of public spaces. The persistent negative correlation
between truck element proportions and safety perception indicates that high
flows of heavy vehicles can cause noise, pollution, and safety risks,
diminishing the appeal and safety of residential environments.

Moreover, increases in the proportions of rider and traffic light elements are
positively correlated with enhanced safety perception, but a threshold effect
exists where further increases do not significantly boost safety perception.
The positive correlation between traffic light proportions and enhanced safety
perception highlights the importance of traffic management facilities in
improving road safety and order. Although increased vegetation proportions are
also positively correlated with enhanced safety perception, in areas with high
vegetation proportions, the positive impact may decrease or even turn
negative.

##  5 Discussion

###  5.1 Automatic Ranking Via MLLMs

The primary findings of this study are verifying the capability of MLLMs to
effectively assess urban safety perception through SVIs. The main advantages
of this technology include enhanced operational automation and efficiency,
ensuring consistency and reproducibility of evaluation results, and the
ability to scale to different cities and broader geographical areas. As more
data accumulates, the precision of safety assessments and the performance of
the model can continuously improve, enabling dynamic updates and iterative
enhancements. However, there are some drawbacks to using MLLMs for urban
safety assessment. Firstly, the training and output of the model could be
influenced by data biases, which might inadvertently reflect these biases in
the results, affecting the fairness of decision-making. The performance
requirements for the model are also very high, for example, only the
proprietary GPT-4V model has achieved satisfactory results. Secondly, an over-
reliance on technology might overlook the value of local knowledge and human
intuition in safety perception. Finally, although the model can provide
quantitative scores, it is often difficult to explain the specific reasons
behind these scores, which limits its application in scenarios requiring high
interpretability.

###  5.2 Overall Urban Analysis

We developed an innovative and efficient methodology that enables global-scale
assessments. This method specifically leverages a sampled anchor set to
extrapolate safety metrics, effectively covering the entire urban landscape.
Unlike previous methods, our approach obviates the need for a training phase,
a significant advantage when data availability is limited. This innovative
methodology not only enhances the precision of safety assessments over large
regions but also significantly improves scalability. As a result, it can be
adapted to a variety of urban settings around the world, providing a robust
tool for urban planners and policymakers to assess and enhance public safety
effectively. However, one notable limitation of our methodology is its
reliance on the pre-trained CLIP model. This dependency poses a challenge when
the safety visual indicators in question exhibit characteristics significantly
distinct from the data used to train the CLIP model. Under such circumstances,
the model’s ability to accurately calculate similarities may be compromised,
leading to ambiguous results.

###  5.3 Potential Applications

The application of MLLMs for automated safety perception assessment holds
profound and extensive potential in urban environments. In the realm of urban
planning and design, MLLMs can swiftly identify high-risk areas, enabling
targeted safety interventions such as improved lighting and optimized public
spaces. By providing quantitative insights into safety perceptions, MLLMs
support data-driven decision-making, thereby justifying and informing urban
development projects. In the field of social research and policy-making, MLLMs
offer invaluable tools for understanding the complex relationships between
perceived safety and various social factors, such as crime rates and
socioeconomic conditions. This technology allows for rigorous evaluation of
policy impacts by comparing safety perception data before and after the
implementation of urban improvement initiatives. Additionally, by
incorporating residents’ feedback into safety assessments, MLLMs foster
community engagement and participatory governance. The integration of MLLMs
also enhances human-machine collaboration by augmenting human capabilities in
processing and analyzing large datasets, ensuring consistent and objective
evaluations. This collaboration enables urban planners, researchers, and
policymakers to focus on strategic decision-making while leveraging the
computational power of MLLMs to derive actionable insights. Given the
performance of MLLMs in substituting human annotation for urban safety
perception tasks, they also hold the potential for automating other urban
perception tasks.

##  6 Conclusion

In this paper, we introduce an innovative automatic pipeline designed to
evaluate perceptions of urban safety. Utilizing advanced MLLMs, our approach
enables an automated safety ranking that eliminates the need for manual human
annotation while maintaining a high correlation with human perception.
Furthermore, we have developed a retrieval-based technique that allows for the
efficient evaluation of SVIs across expansive urban environments. Our findings
reveal significant correlations between the computed safety scores and various
urban elements, suggesting that our model effectively captures the essential
aspects of perceived safety. Such advancements hold promise for facilitating
more adaptive and data-informed urban management strategies, potentially
transforming the landscape of urban governance and planning. Our future plan
aims to expand the capabilities of our pipeline by incorporating real-time
data analysis to dynamically update safety perceptions based on changing urban
conditions. We intend to integrate diverse data sources, including social
media feeds, traffic patterns, and public surveillance footage, to enhance the
accuracy and relevance of our safety assessments.

## Acknowledgment

This work is partly supported by Key Research Base of Humanities and Social
Sciences in Jiangxi Universities 2023 Project JD23003. This work is also
supported by JSPS KAKENHI 24K20795 and World Premier International Research
Center Initiative (WPI), MEXT, Japan.

## References

  * [1] P. Pol, The Safe City: Safety and Urban Development in European Cities, Routledge, New York, 2019.
  * [2] W. Qiu, W. Li, X. Liu, Z. Zhang, X. Li, X. Huang, Subjective and objective measures of streetscape perceptions: Relationships with property value in shanghai, Cities 132 (2023) 104037.
  * [3] L. Wang, X. Han, J. He, T. Jung, Measuring residents’ perceptions of city streets to inform better street planning through deep learning and space syntax, ISPRS Journal of Photogrammetry and Remote Sensing 190 (2022) 215–230.
  * [4] Z. Fan, F. Zhang, B. P. Loo, C. Ratti, Urban visual intelligence: Uncovering hidden city profiles with street view images, Proceedings of the National Academy of Sciences 120 (27) (2023) e2220417120.
  * [5] Y. Yao, Z. Liang, Z. Yuan, P. Liu, Y. Bie, J. Zhang, R. Wang, J. Wang, Q. Guan, A human-machine adversarial scoring framework for urban perception assessment using street-view images, International Journal of Geographical Information Science 33 (12) (2019) 2363–2384.
  * [6] M. P. Salesses, Place pulse: Measuring the collaborative image of the city, Masters thesis, Massachusetts Institute of Technology, Cambridge, MA (2012).
URL https://dspace.mit.edu/handle/1721.1/76533

  * [7] P. Salesses, K. Schechtner, C. A. Hidalgo, The collaborative image of the city: mapping the inequality of urban perception, PloS one 8 (7) (2013) e68400.
  * [8] F. Zhang, B. Zhou, L. Liu, Y. Liu, H. H. Fung, H. Lin, C. Ratti, Measuring human perceptions of a large-scale urban region using machine learning, Landscape and Urban Planning 180 (2018) 148–160.
  * [9] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, J.-R. Wen, A survey of large language models, arXiv preprint arXiv:2303.18223 (2023).
URL http://arxiv.org/abs/2303.18223

  * [10] n. OpenAI, Openai. ”chatgpt.”, https://www.openai.com/research/chatgpt. (2023).
  * [11] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, E. Chen, A survey on multimodal large language models, arXiv preprint arXiv:2306.13549 (2023).
  * [12] OpenAI, Gpt-4 technical report, arXiv (2023).
  * [13] J. Li, D. Li, S. Savarese, S. Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, arXiv preprint arXiv:2301.12597 (2023).
  * [14] K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, Y. Qiao, Videochat: Chat-centric video understanding, arXiv preprint arXiv:2305.06355 (2023).
  * [15] B. Wang, L. Li, Y. Nakashima, H. Nagahara, Learning bottleneck concepts in image classification, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 10962–10971.
  * [16] H. Liu, C. Li, Q. Wu, Y. J. Lee, Visual instruction tuning, Advances in neural information processing systems 36 (2024).
  * [17] D. Zhu, J. Chen, X. Shen, X. Li, M. Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models, arXiv preprint arXiv:2304.10592 (2023).
  * [18] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748–8763.
  * [19] J. Gehl, Life Between Buildings: Using Public Space, Island Press, 2011.
  * [20] Y. Ashihara, The Aesthetic Townscape, MIT Press, 1986.
  * [21] J. Jacobs, The Death and Life of Great American Cities, Random House, 1961.
  * [22] V. Mehta, Evaluating public space, Journal of Urban design 19 (1) (2014) 53–88.
  * [23] K. Mouratidis, The impact of urban tree cover on perceived safety, Urban Forestry & Urban Greening 44 (2019) 126434.
  * [24] E. Zeng, Y. Dong, L. Yan, A. Lin, Perceived safety in the neighborhood: Exploring the role of built environment, social factors, physical activity and multiple pathways of influence, Buildings 13 (1) (2022) 2.
  * [25] O. O. Makinde, The correlates of residents’ perception of safety in gated communities in nigeria, Social Sciences & Humanities Open 2 (1) (2020) 100018.
  * [26] F. Zhang, Z. Fan, Y. Kang, Y. Hu, C. Ratti, “perception bias”: Deciphering a mismatch between urban crime and perception of safety, Landscape and Urban Planning 207 (2021) 104003.
  * [27] Y. Kang, J. Abraham, V. Ceccato, F. Duarte, S. Gao, L. Ljungqvist, F. Zhang, P. Näsman, C. Ratti, Assessing differences in safety perceptions using geoai and survey across neighbourhoods in stockholm, sweden, Landscape and Urban Planning 236 (2023) 104768.
  * [28] Q. Cui, Y. Zhang, G. Yang, Y. Huang, Y. Chen, Analysing gender differences in the perceived safety from street view imagery, International Journal of Applied Earth Observation and Geoinformation 124 (2023) 103537.
  * [29] Y. Liu, M. Chen, M. Wang, J. Huang, F. Thomas, K. Rahimi, M. Mamouei, An interpretable machine learning framework for measuring urban perceptions from panoramic street view images, Iscience 26 (3) (2023).
  * [30] K. Keizer, S. Lindenberg, L. Steg, The spreading of disorder, science 322 (5908) (2008) 1681–1685.
  * [31] G. L. Kelling, J. Q. Wilson, et al., Broken windows, Atlantic monthly 249 (3) (1982) 29–38.
  * [32] J. L. Nasar, The evaluative image of the city, Journal of the American Planning Association 56 (1) (1990) 41–53.
  * [33] D. Halpern, Mental health and the built environment: more than bricks and mortar?, Routledge, 2014.
  * [34] P. Griew, M. Hillsdon, C. Foster, E. Coombes, A. Jones, P. Wilkinson, Developing and testing a street audit tool using google street view to measure environmental supportiveness for physical activity, International Journal of Behavioral Nutrition and Physical Activity 10 (2013) 1–7.
  * [35] N. Naik, J. Philipoom, R. Raskar, C. Hidalgo, Streetscore-predicting the perceived safety of one million streetscapes, in: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2014, pp. 779–785.
  * [36] N. Naik, R. Raskar, C. A. Hidalgo, Cities are physical too: Using computer vision to measure the quality and impact of urban appearance, American Economic Review 106 (5) (2016) 128–132.
  * [37] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems 25 (2012).
  * [38] L. Porzi, S. Rota Bulò, B. Lepri, E. Ricci, Predicting and understanding urban perception with convolutional neural networks, in: Proceedings of the 23rd ACM international conference on Multimedia, 2015, pp. 139–148.
  * [39] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual recognition challenge, International journal of computer vision 115 (2015) 211–252.
  * [40] A. Dubey, N. Naik, D. Parikh, R. Raskar, C. A. Hidalgo, Deep learning the city: Quantifying urban perception at a global scale, in: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, Springer, 2016, pp. 196–212.
  * [41] N. Naik, S. D. Kominers, R. Raskar, E. L. Glaeser, C. A. Hidalgo, Computer vision uncovers predictors of physical urban change, Proceedings of the National Academy of Sciences 114 (29) (2017) 7571–7576.
  * [42] J. Kruse, Y. Kang, Y.-N. Liu, F. Zhang, S. Gao, Places for play: Understanding human perception of playability in cities using street view images and deep learning, Computers, Environment and Urban Systems 90 (2021) 101693.
  * [43] B. Wang, L. Li, M. Verma, Y. Nakashima, R. Kawasaki, H. Nagahara, Mtunet: Few-shot image classification with visual explanations, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, 2021, pp. 2294–2298.
  * [44] Y. Li, N. Yabuki, T. Fukuda, Measuring visual walkability perception using panoramic street view images, virtual reality, and deep learning, Sustainable Cities and Society 86 (2022) 104140.
  * [45] Z. Liu, T. Li, T. Ren, D. Chen, W. Li, W. Qiu, Day-to-night street view image generation for 24-hour urban scene auditing using generative ai, Journal of Imaging 10 (5) (2024) 112.
  * [46] Y. Lu, H.-M. Chen, Using google street view to reveal environmental justice: Assessing public perceived walkability in macroscale city, Landscape and Urban Planning 244 (2024) 104995.
  * [47] Z. Wei, K. Cao, M.-P. Kwan, Y. Jiang, Q. Feng, Measuring the age-friendliness of streets’ walking environment using multi-source big data: A case study in shanghai, china, Cities 148 (2024) 104829.
  * [48] A. Bahrini, M. Khamoshifar, H. Abbasimehr, R. J. Riggs, M. Esmaeili, R. M. Majdabadkohne, M. Pasehvar, Chatgpt: Applications, opportunities, and threats, in: 2023 Systems and Information Engineering Design Symposium (SIEDS), IEEE, 2023, pp. 274–279.
  * [49] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023).
  * [50] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., Palm: Scaling language modeling with pathways, Journal of Machine Learning Research 24 (240) (2023) 1–113.
  * [51] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023).
  * [52] Y. Zhang, P. Liu, F. Biljecki, Knowledge and topology: A two layer spatially dependent graph neural networks to identify urban functions with time-series street view image, ISPRS Journal of Photogrammetry and Remote Sensing 198 (2023) 153–168.
  * [53] R. Herbrich, T. Minka, T. Graepel, Trueskill™: a bayesian skill rating system, Advances in neural information processing systems 19 (2006).
  * [54] J. J. Louviere, T. N. Flynn, A. A. J. Marley, Best-worst scaling: Theory, methods and applications, Cambridge University Press, 2015.
  * [55] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, Y. J. Lee, Llava-next: Improved reasoning, ocr, and world knowledge (2024).
URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

  * [56] M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, et al., Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404.14219 (2024).
  * [57] R. Zhang, J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, Y. Qiao, Llama-adapter: Efficient fine-tuning of language models with zero-init attention, arXiv preprint arXiv:2303.16199 (2023).
  * [58] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, S. Hoi, Instructblip: Towards general-purpose vision-language models with instruction tuning, Advances in Neural Information Processing Systems 36 (2024).
  * [59] L. McInnes, J. Healy, J. Melville, Umap: Uniform manifold approximation and projection for dimension reduction, arXiv preprint arXiv:1802.03426 (2018).