# LM4LV: A Frozen Large Language Model
for Low-level Vision Tasks

Boyang Zheng
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
&Jinjin Gu
Shanghai AI Laboratory
<EMAIL_ADDRESS>
&Shijun Li
Nanjing University
<EMAIL_ADDRESS>
&Chao Dong
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
Shanghai AI Laboratory
<EMAIL_ADDRESS>Work done during internship at Shanghai AI Laboratory

###### Abstract

The success of large language models (LLMs) has fostered a new research trend
of multi-modality large language models (MLLMs), which changes the paradigm of
various fields in computer vision. Though MLLMs have shown promising results
in numerous high-level vision and vision-language tasks such as VQA and text-
to-image, no works have demonstrated how low-level vision tasks can benefit
from MLLMs. We find that most current MLLMs are blind to low-level features
due to their design of vision modules, thus are inherently incapable for
solving low-level vision tasks. In this work, we purpose LM4LV, a framework
that enables a FROZEN LLM to solve a range of low-level vision tasks without
any multi-modal data or prior. This showcases the LLM’s strong potential in
low-level vision and bridges the gap between MLLMs and low-level vision tasks.
We hope this work can inspire new perspectives on LLMs and deeper
understanding of their mechanisms.

##  1 Introduction

The great success and generalizability of Large Language Models (LLMs) have
brought a new research trend of Multi-modality Large Language Models (MLLMs).
We wonder how much LLMs can benefit computer vision to achieve better
performance and realize real intelligence. Recent attempts on MLLMs have
demonstrated promising results on high-level vision tasks, such as image
captioning and visual question answering (VQA). Then we are curious about its
capability on low-level vision tasks, like image denoising and deraining. On
the other hand, since existing works [28, 9] have proved LLMs can already
understand semantic image features, how far are they from directly generating
images as a generative model? All of these converge to the same question: is
it possible to utilize MLLMs to accept, process, and output low-level
features? This is important to further push the limit of MLLMs and low-level
vision. We will make a preliminary exploration in this work.

Existing literature shows a significant gap between MLLMs and low-level vision
tasks. The current major direction for many MLLM related works is towards a
better semantic fusion of the text and image modality. Following this trend,
some works [50, 47] purpose to use LLMs for evaluating low-level attributes
such as lightness and sharpness. However, most low-level vision tasks process
and generate pixel-level information, which does not correspond with
meaningful words. Moreover, the output images must have high fidelity and
consistency with the original images, which is a common flaw of current MLLMs.
Though many works [8, 30, 48, 57, 39, 40, 14, 13] put effort in empowering
MLLMs with image generation capability, most of these MLLMs lack detailed
control of the image. In Sec. 3.1, we show that most MLLMs with image
generation capabilities fail to perform simple image reconstruction. This
indicates that these MLLMs are inherently not capable of handling low-level
vision tasks, as they lack the ability to process low-level details.

Bridging the gap between MLLMs and low-level vision tasks is important for
both fields. MLLMs have shifted the paradigm in many fields of computer vision
by unifying vision tasks into a general conversational manner. However, low-
level vision tasks have not yet benefited significantly from the changes
brought by MLLMs. Currently, most low-level vision modules [8, 45, 22] simply
offer an image-to-image mapping without text interventions. Bridging this gap
could allow us to leverage the strong reasoning and text generation abilities
of MLLMs for low-level vision tasks, providing better user interaction and
greater interpretability in solving low-level vision tasks. On the other hand,
low-level features are important components of images, but are often
overlooked and discarded in current MLLMs. Enabling MLLMs to process low-level
features can lead to a more fine-grained understanding of images and better
control in the process of image generation.

Moreover, the majority of MLLMs are trained on a massive corpus of multimodal
data, with an existing LLM as initialization. However, it is underinvestigated
what the LLM initialization brings to the MLLM. Does the LLM simply offer
strong capabilities for text, or does it also provide underlying abilities for
other modalities as well? Therefore, we highlight the importance of
investigating LLMs’ capability to process visual features with no multi-modal
data or prior, which can lead to a deeper understanding of LLMs’ inner
mechanisms. Although a series of works [23, 28, 51, 31] have put efforts into
investigating the visual feature processing ability of a frozen LLM, none of
them have succeeded in enabling the LLM to produce visual features without
multi-modal supervision. LQAE [23] managed to perform image-to-text tasks in
an in-context learning (ICL) manner without any multi-modal data by quantizing
images into text tokens. However, it fails even for very basic low-level
vision tasks such as deblurring, demonstrating its inability for conditional
image generation.

In this work, we make the first attempt to investigate a frozen LLM’s
capability in processing low-level feature, proving that a frozen LLM can
accept, process and auto-regressively output visual features without any
multi-modal data or prior. By simply training two linear layers with vision-
only data, a frozen LLM showcases non-trivial capability on a wide range of
low-level vision tasks.

##  2 Related Works

###  2.1 Multi-modal Generation with LLMs

Many attempts have been made to equip LLM with multi-modal generation ability.
We categorize these attempts into two types: those that require an additional
text-to-image module and those that do not. The vast majority of research
falls into the former category, such as DreamLLM [8] and KosMOS-G [30], which
necessitate the integration of an external text-to-image module pre-trained
extensively on multi-modal documents, such as Stable Diffusion [11]. In these
instances, the LLM backbone is responsible for producing textual vectors that
serve as the text condition in the text-to-image module, while the actual
conversion from text to image is carried out by a powerful external module. A
significant downside of using overly powerful external text-to-image modules
is their inability to facilitate precise image control, rendering tasks such
as image segmentation and image restoration infeasible. To enhance the
precision of image control, some studies purpose to add skip-connections from
the encoder. PixelLLM [36] employs a lightweight, pixel-level decoder with
skip-connection from encoder to perform language-interactive image
segmentation tasks. SEED-X [14] applies a similar approach by skip-connecting
the original image to the decoder as a reference for conditional image
generation. Though those approaches have shown more precise image control, the
skip-connection raises questions about the role of LLM in the process.

For the latter category, a common architecture is to use a VQGAN [10] to patch
images into tokens. Therefore, the training objective can be unified as next-
token prediction. A number of works [27, 41, 52, 19] have shown the success of
this approach. However, all those model train the whole backbone (either from
scratch or from a existing LLM) on massive multi-modal data. As a result, the
final backbone largely differs from a LLM, failing to provide a clear
understanding of the capability of a LLM in processing visual features.
Therefore, we will not discuss MLLMs of this type in this work. From this
point forward, we will refer MLLM as the first type of MLLM.

###  2.2 Frozen LLM for Tasks of Other Modalities

In contrast to the above-mentioned works that requires heaving LLM backbone
training, some works [28, 23, 51, 31] have shown that a pre-trained LLM
without further training can be used to perform tasks of other modalities. For
example, LimBER [28] shows that training a simple linear layer between a
unsupervised trained vision encoder and a frozen LLM on multi-modal data
empowers the LLM with visual understanding ability. LQAE [23] further shows
the possibility of image-text aligning without multi-modal data. By training a
VQGAN that quantize image to text tokens. LQAE showcases that the LLM is able
to classify image in a in-context-learning (ICL) manner. However, none of them
have succeed to extend the LLM’s capability to generating images. As an
improvement to LQAE, SPAE [51] apply a similar image-to-text-token quantizing
strategy, but introduce CLIP [34] as a cross-modal supervision in VQGAN
training. SPAE is able to encode images directly into semantic-related text
tokens, enabling the LLM for conditional image generation in a ICL manner. So
far, SPAE-like methods has been the only successful approach to enable a
frozen LLM for conditional image generation. However, SPAE still requires a
multi-modal prior (CLIP), and directly encodes images to semantic-related text
tokens. This raise questions about whether LLM is processing visual features
or simply text features. In this work, we show that it’s possible to empower
LLM with conditional image generation ability without any multi-modal data or
prior. This indicates that LLM is able to understand, process and output
visual features even under no image-text alignment.

Figure 1: Reconstruction results of the vision modules in different MLLMs.
Emu2 provides highly semantic consistent images but fails to maintain low-
level details, while MAE can reconstruct images with precise low-level
details.

##  3 Method

###  3.1 Current MLLMs are BLIND to Low-level Features

Our inspiration stems from an observation: current MLLMs [8, 30, 48, 57, 39,
40, 14, 13] with image generation ability are blind to low-level visual
features. Therefore, they are inherently incapable of handling low-level
vision tasks. We attribute the blindness of MLLMs to their vision modules. We
have observed that most MLLMs’ vision modules could not perform image
reconstruction, indicating that the vision encoders in the vision modules are
performing a lossy encoding process. As shown in Fig. 1, the vision module in
MLLMs often tend to capture high-level semantics but fail to maintain low-
level details, conducting a lossy compression of the image. More discussion
and visualization can be found in Appendix A.1.

Currently, most MLLMs follow this lossy compression approach, often relying on
large-scale multi-modal training to train their vision modules. For example,
Emu2 [39] uses 162M image-text pairs to train a vision encoder along with the
LLM backbone and utilizes a pre-trained text-to-image diffusion model [33] as
the decoder. The advantage of this approach is evident: extensive multi-modal
training results in better alignment between vision and text. A better
alignment often brings an improved interaction and fusion of modalities, which
is important for an MLLM. This also allows these MLLMs to use pre-trained LLMs
as the backbone initialization for further training. For instance, SEED [13]
empowers a LLM with image generation ability by fine-tuning the LLM using
LoRA. However, the downside of this alignment is that the visual features
often lose much of the original image information, failing to maintain low-
level details. Therefore, we argue that these MLLMs inherently lack the
ability to handle low-level tasks.

###  3.2 Enable LLM to See Low-level Features

Principles for Choosing Vision Modules. As discussed in Sec. 3.1, current
choices of vision modules in MLLMs fail to maintain low-level details.
Therefore, choosing a suitable vision module that contain full information of
the image is crucial. This allows the LLM backbone to have access to low-level
features, which is the basis for further exploration of the LLM’s capability
in processing low-level features. We argue that two principles are essential
for selecting vision modules. First, the training objective of the vision
module should be reconstruction. This encourages the vision encoder to
maintain low-level details so the encoded feature can be decoded back to pixel
space. Second, the vision modules must be trained in an unsupervised manner to
avoid any multi-modal training. This is important because the LLM already
possesses strong text processing capabilities. If the encoder has already
transformed the image into text-like features, it becomes unclear whether the
LLM is leveraging its powerful text processing abilities to handle text
features or it inherently has the capability to process other modalities.
Under these constraints, only several families of visual encoder remains.
Among them, our final choice is the Masked Autoencoder (MAE) [16], a
representative of the Masked Image Modeling (MIM) family, which encodes an
image to a sequence of visual features and aims to reconstruct the original
image using masked image tokens. We discuss more choices of the vision module
in Sec. 4.3.

Figure 2: Network structure of our design. In the training phase, the visual
tokens and the task tokens learns to prompt the LLM to generate next
visual/text tokens. In the inference phase, the LLM generates visual tokens
and text tokens in an auto-regressive manner. The visual tokens are then
decoded into images.

Fine-tuning MAE for Image Reconstruction. Though MAE is trained to reconstruct
images from a masked token sequence, using MAE directly for image
reconstruction will result in poor performance. We identify that this issue
arises primarily because the released version of MAE calculates the
reconstruction loss solely on masked tokens. This leads to inconsistency
between training and inference behaviors since there are no masked tokens
during image reconstruction. We fine-tune the decoder of MAE on the ImageNet
training set using an L1 reconstruction loss while keeping the encoder frozen.
More details can be found in Appendix A.2.

This fine-tuning significantly improves MAE’s performance in image
reconstruction. Following VQGAN [10, 37], we further incorporate the LPIPS
loss [56] into the training, leading to a better reconstruction FID but a
slightly lower PSNR score. Though [10] shows that a per-patch adversarial loss
may further improve performance, we do not incorporate the adversarial loss as
this may bring artifacts into images [20, 55]. As our objective is to
investigate the capability of LLMs in processing visual features, we expect
the decoder to be faithful to these features, rather than introducing
artifacts during the decoding process.

Table 1: Reconstruction FID (rFID), precision, recall and PSNR on the
validation set of ImageNet. MAE-L1 indicates to use L1 loss for fine-tuning
MAE’s decoder. MAE* is the version tuned by a combination of L1 loss and LPIPS
Loss. Best results are bolded.

Model |  rFID$\downarrow$ |  prec(%)$\uparrow$ |  recall(%)$\uparrow$ |  PSNR$\uparrow$
---|---|---|---|---
MAE | 84.22 | 13.35 | 45.78 | 19.15
MAE-L1 | 9.96 | 88.46 | 97.57 | 29.21
VQGAN | 1.49 | 94.90 | 99.67 | 22.61
MAE* | 1.24 | 99.94 | 99.97 | 28.96

We compare MAE with a commonly used image reconstruction module VQGAN111We use
a VQGAN from https://github.com/CompVis/taming-transformers, trained on
OpenImage with downsample rate 8 and a vocabulary of 16384 [10]. From Tab. 1
and Fig. 4, the fine-tuned MAE has the best reconstruction ability,
introducing less artifacts and showing better face reconstruction. More
training details can be seen in Appendix A.2. From this point forward, unless
specified otherwise, the term MAE refers to the fine-tuned version MAE*.

###  3.3 Next Element Prediction on Low-level Vision Tasks

Similar to Emu and Emu2 [40, 39], we apply a next element prediction strategy
to enable LLMs to accept visual features and output visual features in an
auto-regressive manner. Our main network structure is illustrated in Fig. 2.

Adapter Modules. As we want to investigate the LLM’s capability to process
low-level vision features, we need to constrain the adapter’s complexity.
Therefore, we use two simple linear layers as the adapter modules between the
LLM and the vision encoder/decoder to align the feature dimension. We call the
visual features "visual tokens" if they are aligned with the LLM’s dimension.

Training & Generation Scheme. In training, we apply the standard next-token
cross-entropy loss for text tokens. For continuous visual tokens, we apply a
next token $l_{2}$-regression loss. This unifies the training process for both
text and visual as next-element prediction tasks. For generation, we use the
auto-regressive generation scheme, generating one text or visual token at a
time. To switch between generating visual and text tokens, we set that text
tokens are generated by default, and visual features are delineated by markers
 and  before and after, respectively. Generation of a visual
feature occurs only after the LLM has generated . After generating a
sequence of visual tokens, the LLM returns to generating text tokens. The
visual token sequence is then transformed to visual features through the
linear adapter and decoded into an image by the visual decoder.

Under this paradigm, we define low-level vision tasks as conditional image
generation tasks performed under visual instructions. For a pair consisting of
a low-quality image and a high-quality image, we convert this into a
conversation format:

Human: <LQ-image> Assistant: <HQ-image>

Here <LQ-image> and <HQ-image> represent the visual feature sequences (after
linear projection) for low and high quality images respectively. We apply an
instruction-tuning strategy, similar to [24], only calculating the loss for
the desired output <HQ-image>.

However, the current design lacks a description of the target low-level vision
tasks. This may lead to the inclusion of task-related soft prompts within the
trained adapter projection, which is not desired as we expect the adapter
module to purely focus on the translation between image and text space. Since
low-level vision tasks are challenging to be described precisely using
language, we incorporate a trainable task token sequence <task> within the
instructions. This serves as a soft prompt to guide the LLM in performing
specific low-level vision tasks. The final data format is as follows:

Human: <LQ-image> <task> Assistant: <HQ-image>

In this format, normal text tokens (marked in black) is tokenized normally.
The visual tokens (marked in blue) are encoded by a vision encoder and a
linear layer. The task token (marked in red) is a trainable embedding sequence
that is directly inserted into the input sequence. Within the entire pipeline,
the only trainable parameters are the two linear adaptation modules and the
task token sequence, which are jointly optimized during training.

##  4 LLM’s Capability on Low-level Tasks

###  4.1 Experiment Setup

Base Models & Hyperparameters. We use LLaMA2-7B
instruct222https://llama.meta.com/llama2/ [43] as the base LLM for all our
experiments. For vision modules, we use MAE-
Large333https://huggingface.co/facebook/vit-mae-large [16] and fine-tune the
decoder as mentioned in Sec. 3.2. For adaptation modules, we only use linear
layers as an affine transformation. We use a trainable task token sequence of
length 10 by default.

Training Details. We use the
LLAVA595k444https://huggingface.co/datasets/liuhaotian/LLaVA-
CC3M-Pretrain-595K [24] dataset as our base dataset for degradation generation
without data augmentation. All images are resized to $224\times 224$ to fit
the input size of MAE. We use an actual batchsize of $256$. By default we
train the model for 2 epochs as we observe convergence after 2 epochs. We use
AdamW as the optimizer, with $\beta=(0.9,0.95),lr=3\times 10^{-4}$ and no
weight decay. We use warm-up decay strategy with 200 warm-up steps. All
experiments are done on a maximum of 4 NVIDIA A100 GPUs. A single training
takes around 8 hours.

Evaluation tasks. We evaluate our method on several representative low-level
vision tasks: denoising [46, 53, 54], deblurring [1, 6], pepper noise removal
[12, 15], deraining [5, 25, 49] and mask removal [26, 44]. We use the NoCaps
dataset555https://huggingface.co/datasets/HuggingFaceM4/NoCaps as the test set
and add the same degradations as training. We use PSNR and SSIM as our default
metrics. Additionally, SPAE [51] shows that a simply rotation could be
challenging under a similar setup. Thus we set two simple tasks that requires
large spatial operation: image rotation and image flipping. We name the tasks
as restoration tasks and spatial operation tasks respectively.

Degradation details. For denoising, We add Gaussian noise with zero mean and a
random standard variance uniformly sampled from $[0,50/255]$. For deblurring,
we add Gaussian blur to the images with a window size uniformly sampled from
$\\{1,3,5,7\\}$. For deraining, we add rains at random angles and positions to
the images with an amount uniformly sampled from $[0,20]$. For pepper noise
removal, we set the portion of peppers in the degraded images to be uniformly
sampled from $[0,0.1]$. For mask removal, we use a mask size of $4$ and a
masking rate of $0.1$.

Baselines. As our goal is to investigate whether LLM has the ability to
process low-level features and handle low-level vision tasks, we only need to
verify that using LLM is better than doing nothing. Thus we set a baseline of
using MAE to reconstruct degraded images without further modification (denote
as MAE-r). In restoration tasks, this baseline serves as a lower-bound for
MAE-based image restoration methods. In spatial operation tasks, this baseline
serves as the upper-bound, as the degraded images in this case is the operated
images.

###  4.2 LLM Shows Non-trivial Capability on Low-level Vision Tasks

Our main results are shown in Tab. 2, with some visualizations presented in
Fig. 3. From Tab.2, LM4LV stably obtains a higher PSNR and SSIM score than the
MAE-r baseline among all restoration tasks. Under image denoising task, LM4LV
achieves a higher PSNR score with an increase of 6.81dB (from 19.96dB to
26.77dB). On all restoration tasks, LM4LV achieves an average PSNR score
increase of 3.96dB, an average of SSIM increase of 0.09. On spatial operation
tasks, LM4LV achieves high PSNR and SSIM, enclosing the margin to the upper-
bound baseline. The results indicates that LLM has non-trivial capability in
processing and outputting raw visual features. We give more visualizations and
failure cases in Appendix C.1 and Appendix C.2.

Table 2: Results of LM4LV on various low-level vision tasks. The top five tasks are image restoration tasks, the bottom two tasks do not require restoration, but involve large-scale spatial operations. Tasks | Degraded | MAE-r | LM4LV
---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ |  |  $\Delta$PSNR/SSIM
---
Denoising | 23.11dB | 0.49 | 19.96dB | 0.65 | 26.77dB | 0.80 | +6.81dB/+0.15
Deblurring | 30.88dB | 0.83 | 26.14dB | 0.78 | 26.23dB | 0.79 | +0.09dB/+0.01
Deraining | 20.52dB | 0.84 | 19.96dB | 0.74 | 24.62dB | 0.77 | +4.66dB/+0.03
Pepper Removal | 19.22dB | 0.51 | 23.01dB | 0.58 | 25.20dB | 0.75 | +2.19dB/+0.17
Mask Removal | 20.54dB | 0.83 | 20.00dB | 0.73 | 25.83dB | 0.80 | +5.83dB/+0.07
Rotation |  inf666Some implementation will return a PSNR score of 100 when two images are identical. We stick to the mathematical definition of PSNR here and return infinite PSNR score when two images are the same. | 1.00 | 29.52dB | 0.89 | 27.18dB | 0.83 | -2.34dB/-0.06
Flipping | inf | 1.00 | 29.52dB | 0.89 | 27.28dB | 0.84 | -2.24dB/-0.05
Figure 3: A frozen LLM shows non-trivial capability on various low-level
vision tasks.

###  4.3 Choice of Vision Module Matters

The key component in our method is the visual module. While we have
successfully showed LLM’s low-level visual feature processing ability using a
fine-tuned MAE, it prompts an investigation into how different visual modules
might affect the outcomes.

There are actually not many choices of vision modules for unsupervised image
reconstruction. A common choice is VQ-GAN, a state-of-the-art representative
of the VAE family. Another less common choice is BEiT [3], which is originally
used for image recognition. The training goal of BEiT involves predicting the
masked tokens of the DALL-E tokenizer777OPENAI public release at
https://cdn.openai.com/dall-e/decoder.pkl &
https://cdn.openai.com/dall-e/encoder.pkl [35], which can be decoded into
images by the DALL-E’s decoder. In practice, we find that BEiT’s predictions
are accurate enough to roughly reconstruct images even without fine-tuning,
potentially due to a lower masking rate during training compared to MAE.

We use MAE, VQGAN and BEiT as the vision module in our pipeline and evaluate
their performance. We start with a trivial task: image repetition. The LLM is
asked to repeat the input image. As shown in Fig. 4, all three modules succeed
in performing identity mapping. However, asked for a slightly more complex
task: image rotation, VQGAN and BEiT produces messy images with no semantic
meaning, while MAE still performs well. This indicates that the choice of
vision module is important for the success of our method. More discussion can
be found in Appendix A.3.

Figure 4: All three modules succeed in performing image repetition, but VQGAN
and BEiT totally fail for image rotation.

###  4.4 Auto-regressive Generation Matters

Another question to consider is the necessity of auto-regressive (AR)
generation. Indeed, AR is not the mainstream approach for image generation.
Current AR methods [21, 7, 29, 10] often underperform diffusion-based methods
[32, 11] and tend to require higher computational costs. Therefore, we design
a more straightforward, more vision-style generation scheme. We feed the
degraded image tokens to the LLM and expect it to directly output curated
image tokens in a single forward process. We call this generation process
"ViT-LLM generation", as it treats LLM as a normal ViT. We still use the
$l_{2}$-regression on visual features as the training objective.

In the training process, the trainable parameters in this process are two
linear adaptation layers. Furthermore, we cancel the causal attention mask and
the ROPE position embedding [38] in the forward process, as they are not the
common practice for vision modules. We test this generation scheme on image
denoising, under the experiment setup mentioned in Sec. 4.1. As shown in Fig.
5, ViT-LLM generation produces low-quality and blurred images, even when the
noise level is low. This indicates auto-regressive generation is essential for
our success. Moreover, using auto-regressive feature generation naturally
aligns with LLM’s behavior, and can be seamlessly plugged into LLM’s
generation as an additional encoding and decoding module.

Figure 5: ViT-LLM generation fails for image denoising even when the noise
level is low (2nd row), producing low-quality and blurred images.

##  5 Abalation Studies

In order to ensure that it is the LLM rather than other modules that plays the
crucial role in processing low-level features, we have intentionally
simplified the design of other components. However, we still require extensive
ablation study to further validate the importance of the LLM. We also
investigate a single LLM layer’s capacity in Appendix B.3 and explore more
base LLMs in Appendix B.2.

###  5.1 Is the Linear Layer Doing the Task?

Although our adaptation modules are intentionally simplified to a simple
linear layer, we still need to verify whether it is the adaptation module that
accomplishes the low-level vision tasks. To this end, we remove the LLM
component and the auto-regressive generation process from the model, leaving
only the linear adaptation module. At this stage, the training objective is
use a linear layer to map the low-quality visual features to high-quality
visual features produced by MAE.

We train the linear layer using $l_{2}$ regression for image denoising, under
the same setup as described in Sec. 4.1. Some visualizations are shown in Fig.
6. It is evident that a single linear layer is insufficient to handle low-
level vision tasks effectively. Though the main structure of the images
remains, the images have weird colors and are divided into patches.

Additionally, we observe that through training, two linear layers tend to
perform a scaled identity mapping even though they are not forced to do so.
This further evidences the importance of LLM in processing visual features.
More analysis can be found in Appendix B.1.

Figure 6: Using a single linear layer for denoising yields bad results.

###  5.2 Does Text Pre-training Play an Important Role?

Some studies [4, 2] have found that a randomly initialized CNN module can
already serve as an effective representation extractor, suggesting that the
architecture of the model itself possesses the ability to solve various tasks.
Thus, a natural question arises: Does text pre-training endow the LLM with the
ability to solve low-level vision tasks, or is the capability inherent to the
LLM’s architecture itself? [2] requires numerous random initializations are
required to define kernels for classification purposes, yet the integration of
multiple random initializations with auto-regressive generation on an LLM has
not been discussed. Therefore, we only initialized the LLM randomly once and
maintained the architecture unchanged, testing its denoising performance under
the setup in 4.1. Visualizations from Fig. 7 demonstrate that the randomly
initialized LLM fails to generate meaningful images, indicating that text pre-
training is crucial for the LLM to solve low-level vision tasks.

Figure 7: Using randomly initialized LLM gives messy outputs.

###  5.3 LLM vs Expert Models

Although we have validated the capability of LLMs to process visual features,
the question remains, by how much? To quantitatively assess the capability of
LLMs, we replaced the LLM with either an 2-layer MLP or a one-layer
Transformer, serving as expert models to solve visual tasks. To ensure fair
comparisons, the MLP and Transformer have nearly the same number of trainable
parameters as our method. For the expert models, we adopted the same design
described in Sec. 5.1, replacing the linear layer with the expert model.

We test the performance of expert models on image denoising and image
rotation, under the exact same setup as mentioned in Sec. 4.1. From Table 3,
our method surpasses the MLP baseline but falls behind the Transformer
baseline in image denoising tasks. But both MLP and Transformer fail to do
image rotation, whereas LLM can handle it well. This proves LLMs’ non-trivial
low-level feature processing capability.

Table 3: Comparisons of different expert models and our methods. Using LLM
gain superior performance in image rotation, and surpass MLP in image
denoising. Best results are in bold.

| Denoising | Rotation
---|---|---
|  PSNR$\uparrow$ |  SSIM $\uparrow$ |  PSNR$\uparrow$ |  SSIM $\uparrow$
MLP | 25.87dB | 0.76 | 13.29dB | 0.32
Transformer | 27.42dB | 0.81 | 10.52dB | 0.23
Ours* | 26.77dB | 0.80 | 27.18dB | 0.83

###  5.4 Discussion & Limitations

Discussion. Our work could lead to some interesting topic beyond enabling LLMs
for low-level vision tasks. As stated, the goal for this work is not to
achieve the best performance in image restoration, but to demonstrate the
potential of LLMs in processing low-level features and show a possible way to
leverage LLMs’ strong reasoning and interaction ability to low-level vision
tasks. Also, as LM4LV does not involve any multi-modal data, this framework
could be extended beyond vision to the fields where cross-modal data is scarce
by replacing MAE with a self-supervised field-specific module.

Limitations. As shown in Fig. 3, LM4LV could not restore high-frequency
details in degraded images. This is natural because the LLM does not have
image prior, which could be improved by adding skip-connection or multi-modal
data. But this is not the focus of this work. Also, the performance gap
observed between our method and a one-layer Transformer also indicates that
there is room for improvement in our approach.

##  6 Conclusion

In this work, we aim to answer the question: Does a frozen LLM has the ability
to accept, process, and output low-level features? By designing a framework
from bottom to top, we give a positive answer, showing LLMs’ non-trivial
performance on various low-level tasks. We hope this work can inspire new
perspectives on the capabilities of LLMs and deeper understanding of their
mechanisms.

## References

  * Abuolaim and Brown [2020] Abdullah Abuolaim and Michael S Brown.  Defocus deblurring using dual-pixel data.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16_ , pages 111–126. Springer, 2020.
  * Amid et al. [2022] Ehsan Amid, Rohan Anil, Wojciech Kotłowski, and Manfred K. Warmuth.  Learning from Randomly Initialized Neural Network Features, February 2022.  URL http://arxiv.org/abs/2202.06438.
  * Bao et al. [2022] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  BEiT: BERT Pre-Training of Image Transformers, September 2022.  URL http://arxiv.org/abs/2106.08254.
  * Cao and Wu [2021] Yun-Hao Cao and Jianxin Wu.  A Random CNN Sees Objects: One Inductive Bias of CNN and Its Applications, December 2021.  URL http://arxiv.org/abs/2106.09259.
  * Chen et al. [2021a] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.  Pre-trained image processing transformer.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12299–12310, 2021a.
  * Chen et al. [2021b] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen.  Hinet: Half instance normalization network for image restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 182–192, 2021b.
  * Chen et al. [2020] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _International conference on machine learning_ , pages 1691–1703. PMLR, 2020.
  * Dong et al. [2024] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi.  DreamLLM: Synergistic Multimodal Comprehension and Creation, March 2024.  URL http://arxiv.org/abs/2309.11499.
  * Eichenberg et al. [2022] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank.  MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning, October 2022.  URL http://arxiv.org/abs/2112.05253.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming Transformers for High-Resolution Image Synthesis, June 2021.  URL http://arxiv.org/abs/2012.09841.
  * Esser et al. [2024] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach.  Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, March 2024.  URL http://arxiv.org/abs/2403.03206.
  * Fu et al. [2019] Bo Fu, Xiaoyang Zhao, Chuanming Song, Ximing Li, and Xianghai Wang.  A salt and pepper noise image denoising method based on the generative classification.  _Multimedia Tools and Applications_ , 78(9):12043–12053, 2019.
  * Ge et al. [2023] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan.  Planting a SEED of Vision in Large Language Model, August 2023.  URL http://arxiv.org/abs/2307.08041.
  * Ge et al. [2024] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan.  SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation, April 2024.  URL http://arxiv.org/abs/2404.14396.
  * Gebreyohannes and Kim [2012] Tina Gebreyohannes and Dong-Yoon Kim.  Adaptive noise reduction scheme for salt and pepper.  _arXiv preprint arXiv:1201.2050_ , 2012.
  * He et al. [2021] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked Autoencoders Are Scalable Vision Learners, December 2021.  URL http://arxiv.org/abs/2111.06377.
  * Hessel et al. [2022] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: A Reference-free Evaluation Metric for Image Captioning, March 2022.  URL http://arxiv.org/abs/2104.08718.
  * Huh et al. [2024] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola.  The Platonic Representation Hypothesis, May 2024.  URL http://arxiv.org/abs/2405.07987.
  * Jin et al. [2024] Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, Di Zhang, Wenwu Ou, Kun Gai, and Yadong Mu.  Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization, March 2024.  URL http://arxiv.org/abs/2309.04669.
  * Ledig et al. [2017] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network, May 2017.  URL http://arxiv.org/abs/1609.04802.
  * Lee et al. [2022] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han.  Autoregressive Image Generation using Residual Quantization, March 2022.  URL http://arxiv.org/abs/2203.01941.
  * Lin et al. [2024] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Wanli Ouyang, Yu Qiao, and Chao Dong.  DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior, April 2024.  URL http://arxiv.org/abs/2308.15070.
  * Liu et al. [2023a] Hao Liu, Wilson Yan, and Pieter Abbeel.  Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment, February 2023a.  URL http://arxiv.org/abs/2302.00902.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual Instruction Tuning, December 2023b.  URL http://arxiv.org/abs/2304.08485.
  * Liu et al. [2022] Lin Liu, Lingxi Xie, Xiaopeng Zhang, Shanxin Yuan, Xiangyu Chen, Wengang Zhou, Houqiang Li, and Qi Tian.  Tape: Task-agnostic prior embedding for image restoration.  In _European Conference on Computer Vision_ , pages 447–464. Springer, 2022.
  * Liu et al. [2023c] Yihao Liu, Xiangyu Chen, Xianzheng Ma, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong.  Unifying image processing as visual prompting question answering.  _arXiv preprint arXiv:2310.10513_ , 2023c.
  * Lu et al. [2023] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi.  Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action, December 2023.  URL http://arxiv.org/abs/2312.17172.
  * Merullo et al. [2023] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick.  Linearly Mapping from Image to Text Space, March 2023.  URL http://arxiv.org/abs/2209.15162.
  * Nash et al. [2021] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia.  Generating images with sparse representations.  _arXiv preprint arXiv:2103.03841_ , 2021.
  * Pan et al. [2024] Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei.  Kosmos-G: Generating Images in Context with Multimodal Large Language Models, March 2024.  URL http://arxiv.org/abs/2310.02992.
  * Pang et al. [2024] Ziqi Pang, Ziyang Xie, Yunze Man, and Yu-Xiong Wang.  Frozen Transformers in Language Models Are Effective Visual Encoder Layers, May 2024.  URL http://arxiv.org/abs/2310.12973.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable Diffusion Models with Transformers, March 2023.  URL http://arxiv.org/abs/2212.09748.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, July 2023.  URL https://arxiv.org/abs/2307.01952v1.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision, February 2021.  URL http://arxiv.org/abs/2103.00020.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-Shot Text-to-Image Generation, February 2021.  URL http://arxiv.org/abs/2102.12092.
  * Ren et al. [2023] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin.  PixelLM: Pixel Reasoning with Large Multimodal Model, December 2023.  URL http://arxiv.org/abs/2312.02228.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-Resolution Image Synthesis with Latent Diffusion Models, April 2022.  URL http://arxiv.org/abs/2112.10752.
  * Su et al. [2023] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  RoFormer: Enhanced Transformer with Rotary Position Embedding, November 2023.  URL http://arxiv.org/abs/2104.09864.
  * Sun et al. [2024a] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Generative Multimodal Models are In-Context Learners, May 2024a.  URL http://arxiv.org/abs/2312.13286.
  * Sun et al. [2024b] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Emu: Generative Pretraining in Multimodality, May 2024b.  URL http://arxiv.org/abs/2307.05222.
  * Team [2024] Chameleon Team.  Chameleon: Mixed-Modal Early-Fusion Foundation Models, May 2024.  URL http://arxiv.org/abs/2405.09818.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023.  URL http://arxiv.org/abs/2307.09288.
  * Wang et al. [2018] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In _Proceedings of the European conference on computer vision (ECCV) workshops_ , pages 0–0, 2018.
  * Wang et al. [2021a] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan.  Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data, August 2021a.  URL http://arxiv.org/abs/2107.10833.
  * Wang et al. [2021b] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li.  Uformer: A General U-Shaped Transformer for Image Restoration, November 2021b.  URL http://arxiv.org/abs/2106.03106.
  * Wu et al. [2023a] Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al.  Q-instruct: Improving low-level visual abilities for multi-modality foundation models.  _arXiv preprint arXiv:2311.06783_ , 2023a.
  * Wu et al. [2023b] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  NExT-GPT: Any-to-Any Multimodal LLM, September 2023b.  URL http://arxiv.org/abs/2309.05519.
  * Yang et al. [2017] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan.  Deep joint rain detection and removal from a single image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1357–1366, 2017.
  * You et al. [2024] Zhiyuan You, Zheyuan Li, Jinjin Gu, Zhenfei Yin, Tianfan Xue, and Chao Dong.  Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models, March 2024.  URL http://arxiv.org/abs/2312.08962.
  * Yu et al. [2023a] Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David A. Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, Kevin Murphy, Alexander G. Hauptmann, and Lu Jiang.  SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs, October 2023a.  URL http://arxiv.org/abs/2306.17842.
  * Yu et al. [2023b] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan.  Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning, September 2023b.  URL http://arxiv.org/abs/2309.02591.
  * Zhang et al. [2017a] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  _IEEE transactions on image processing_ , 26(7):3142–3155, 2017a.
  * Zhang et al. [2017b] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep cnn denoiser prior for image restoration.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3929–3938, 2017b.
  * Zhang et al. [2020] Kai Zhang, Luc Van Gool, and Radu Timofte.  Deep Unfolding Network for Image Super-Resolution, March 2020.  URL http://arxiv.org/abs/2003.10428.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, April 2018.  URL http://arxiv.org/abs/1801.03924.
  * Zhu et al. [2023] Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan.  VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation, December 2023.  URL http://arxiv.org/abs/2312.09251.

##  Appendix A More Implementation and Experiment Details

###  A.1 MLLMs’ Inability to See Low-level Visual Features

In this section we give more detail about how MLLMs fail to process low-level
visual features. As shown in Tab. 4, we group MLLMs with conditional image
generation ability into three categories based on the type of vision embedding
they use: token, unknown, and feature. We show that our method LM4LV is
capable of image reconstruction, while have no need any heavy pre-training and
any multi-modal data.

For closed-source models with no access to the vision module, we test a
similar task: image repetition. That is, we prompt the model to repeat the
input image without any modification. We show that Gemini [42], GPT-4V and
GPT-4o fail to repeat the input image, giving highly semantic related images
that are different in low-level details. We give the visualizations of image
repetition tasks in Fig. 8. But we would like to note this is a naive approach
to test the capability of vision modules, and the results could be improved
with more detailed and sophisticated prompts.

Table 4: Most MLLMs with conditional image generation ability fails for image
reconstruction. Listed are MLLMs that unify comprehension and generation,
grouped by vision embedding type. “Features” means continuous features,
“Token” represents discrete tokens. Multi-modal backbone pre-training
represents that the model needs a large corpus of multi-modal data for pre-
training. Multi-modal vision encoding/decoding indicates that the vision
encoder/decoder is trained under multi-modal data or supervised by additional
multi-modal modules. N/A means we have no knowledge of the vision module.

| Vision
---
Embedding
Type
| Model
---
| Conditional
---
Image
Generation
| Multi-modal
---
Backbone
Pre-training
| Multi-modal
---
Vision Encoding
| Multi-modal
---
Vision Decoding
| Open-
---
source
| Image
---
Reconstruction
Token | SEED-OPT | $\checkmark$ | LoRA | $\checkmark$ | $\checkmark$ | $\times$ | $\times$
LaVIT | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$
SEED-LLaMA | $\checkmark$ | LoRA | $\times$ | $\checkmark$ | $\checkmark$ | $\times$
Unknown | Gemini | $\checkmark$ | $\checkmark$ | N/A | N/A | $\times$ | $\times$
GPT-4V | $\checkmark$ | $\checkmark$ | N/A | N/A | $\times$ | $\times$
GPT-4o | $\checkmark$ | $\checkmark$ | N/A | N/A | $\times$ | $\times$
Feature | Emu | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$
VL-GPT | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$
DreamLLM | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$
Emu2 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$
NExT-GPT | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$
SEED-X888SEED-X is capable of image reconstruction if the vision decoder is provided with the original images. But here we adopt the w/o original image setting. | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$
LM4LV(ours) | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$

Figure 8: Image repetition results of Gemini, GPT-4V and GPT-4o. The models
fail to repeat the input image, giving highly semantic related images that are
different in low-level details. We note that the results could be improved
with more detailed and sophisticated prompts.

###  A.2 Fine-tuning Details of MAE

As shown in Fig. 10, directly using MAE for image reconstruction results in
poor performance. No previous work has demonstrated the capability of MAE for
image reconstruction, and through our experiments, we found that MAE is
capable of reconstructing images if fine-tuned properly. To ensure the
representation of the visual features is not changed during fine-tuning, we
freeze the encoder and only train the decoder. Given an image $x$ and it’s MAE
reconstruction $\tilde{x}$, we use a reconstruction L1 loss and a LPIPS loss
as the training objective, defined as follows:

$\mathcal{L}=\left|x-\tilde{x}\right|+\lambda\cdot\text{LPIPS}(x,\tilde{x})$
(1)

Following the training recipe in VQGAN [10], we set $\lambda=1$. Following the
pre-training recipe of MAE, we use an actual batchsize of $4096$ and a basic
learning rate of $1e-4$. The actual learning rate is scaled by batchsize /
256. We use the AdamW optimizer with $\beta=(0.9,0.95)$ and a weight decay of
$0$. We use warm-up decay scheduling with a warm-up step of $100$. We train
the model on ImageNet for 28 epochs without data augmentation. All images are
resized to $224\times 224$, matching the input size of MAE. Though we use MAE
to model degradations in our experiment, we do not include degraded images
into the training data. This further evidences MAE’s strong generality.

Additionally, we would like to note that LPIPS requires a VGG model as
supervision, which is trained on image-label data pairs. However, we re-state
that the encoder is frozen during the fine-tuning process. Thus the encoder is
still trained in an unsupervised manner, and the LLM only have access to the
unsupervised visual features. Thus, though the decoder is trained under
slightly supervised manner, it does not affect our conclusion that the LLM is
capable of processing and outputting unsupervised visual features. Moreover,
only using L1 loss as the training objective still achieves a good performance
as shown in Tab. 1. We visualize the reconstruction results of MAE fine-tuned
under different objectives in Fig. 10.

Another interesting topic is how fine-tuning effects the original MAE’s
capability i.e. reconstructing masked tokens. As shown in Fig. 9, the fine-
tuned MAE totally loss the capability of reconstructing masked tokens,
producing messy patches for them. However, MAE is still able to reconstruct
the unmasked tokens, indicating that the fine-tuned MAE decoder has a strong
locality.

Figure 9: The fine-tuned MAE decoder tends to have a strong locality. We mask
the diagonal image tokens and use MAE for reconstruction. The fine-tuned MAE
is able to reconstruct the unmasked tokens well, but fails to reconstruct the
masked tokens. Figure 10: Directly using MAE for reconstruction gives bad-
quality images. Using L1-loss to fine-tune MAE gives a much better
performance. The LPIPS loss further improves the quality of the reconstructed
images.

###  A.3 Choice of vision modules

VQGAN We use a publicly available VQGAN from [10], with a downsample rate of
16 and a vocabulary size of 16384. The state-of-the-art VQVAE has $32\times
32=1024$ tokens, which is way larger than MAE (197) and BEiT (197), thus we
use a version with inferior performance but much fewer tokens ($16\times
16=256$) for a faster training and fairer comparison.

A VQ-GAN will first encodes images into 2D feature matrixs. We flatten the
feature into 1D in raster scan order to obtain visual tokens.

BEiT We use BEiT-
Large999https://github.com/addf400/files/releases/download/v1.0/beit_large_patch16_224_pt22k.pth,
trained under ImageNet22k in an unsupervised manner. BEiT encodes an image to
197 tokens, including a CLS token. Though the DALL-E decoder does not need CLS
token for decoding, we still add CLS token into visual tokens as CLS may guide
the LLM for later generation. However, two factors limit BEiT to be a good
model for image reconstruction. First, BEiT uses DALL-E as the tokenizer,
which is relatively bad at image reconstruction. DALL-E owns a reconstruction
FID of 32.01 on Imagenet validation set, which is ten times higher than the
VQGAN we use. Secondly, BEiT uses a input image size of $112\times 112$ for
DALL-E, which is different from the training image size of DALL-E ($256\times
256$). This further hurts the performance of BEiT in image reconstruction.

As shown in Sec. 4.3, using VQGAN and BEiT as the vision module in our
pipeline fails even for image rotation. Why is it? A very recent work [18]
purpose a hypothesis that models of different modalities are learning similar
representations, and the similarity can be quantified using pre-defined kernel
distance. They show a stronger vision learner learns more similar
representation to that of LLMs. From the perspective of linear probing, MAE is
definitely a much stronger vision learner than VQGAN and BEiT. Thus, MAE may
be more aligned with LLMs. Thus LLM is more easy to process the visual
features of MAE than that of VQGAN and BEiT. We further side-prove this
alignment using vision-language tasks.

We freeze the vision encoder and the LLM, and train a linear layer to project
the visual features into the LLM’s feature space under image captioning tasks.
The training setup is identical to the first stage of LLAVA [24], except we
choose a 100k subset of LLAVA595k for quicker training. We then test the
ClipScore and Ref-ClipScore [17] on the NoCaps dataset. As shown in Tab. 5,
MAE has a far higher ClipScore and Ref-ClipScore than BEiT, while BEiT is
slightly better than VQGAN. This could be a possible explanation for the
failure of VQGAN and BEiT in our pipeline.

Table 5: ClipScore and Ref-ClipScore of different vision encoders. Random
represents to use 10 random letters as the generated caption.

|  CS$\uparrow$ |  Ref-CS $\uparrow$
---|---|---
Random | 34.38 | 46.01
BEiT | 37.52 | 47.78
MAE | 46.97 | 55.56

##  Appendix B Analysis and Discussion

###  B.1 Linear Adapters Tend to Perform a Identity Mapping

We found that though not explicitly trained to do so, the linear layers in LLM
tend to perform a scaled identity mapping $\alpha I,\alpha\in R$. Fig 11 shows
the visualized result of the multiplication of two weight matrix in the linear
layers. It can be seen that the weight of the results centers mostly on the
diagonal of the matrix. We further numerically analyze how close the matrix is
to a scaled identity matrix. Specifically, we define two metric as a way to
measure the similarity between a matrix $A$ and an scaled identity matrix
$\alpha I,\alpha\in R$:

$\left\\{\begin{aligned} \mathcal{L}(A)&=\min_{\alpha}\|A-\alpha
I\|^{2}_{2}\\\ \mathcal{D}(A)&=\frac{1}{n(n-1)}\sum_{i\neq
j,i,j\in[1,n]}\left|\frac{A_{i,i}}{A_{i,j}}\right|\end{aligned}\right.$

From Tab. 6, the linear layers obtained by training have a $\mathcal{L}$ and
$\mathcal{D}$ very similar to a identity matrix. This indicates the linear
layers tends to be a scaled identity mapping even though they are not forced
to do so, which further evidences the importance of LLM in processing visual
features.

Table 6: Our multiplication matrix have $\mathcal{L},\mathcal{D}$ similar to that of an Identity matrix. Random represents a randomly initialized weight subject to standard Gaussian distribution. Identity represents an identity mapping.  | Random | Ours | Identity
---|---|---|---
$\mathcal{L}$ | $1.00$ | $5.26\times 10^{-4}$ | $0$
$\mathcal{D}$ | $12.52$ | $1.53\times 10^{4}$ | inf

Figure 11: The multiplication matrix tends to center it’s weight on the
diagonal. Yellow represents a large value, and blue represents a small value.

Below we first clarify our definition of $\mathcal{L}$ and $\mathcal{D}$, and
then provide an analysis of the expectation of $\mathcal{L},\mathcal{D}$ on a
randomly initialized matrix $N^{n\times n}$.

Clarification. Ideally, we expect the linear layer to do nothing except for a
simply identity mapping with a constant scaling(with a constant shift brought
by the bias). That is, we want the multiplication matrix $A$ to be close to
$\alpha I$, where $\alpha$ is a constant and $I$ is the identity mapping. If
so, the linear layer will not change the input features(except for a scaling),
and we can prove that it is LLM that is processing the visual features.
Therefore, $\mathcal{L}(A)$ is defined as a minimum l2-distance between $A$
and a scaled identity mapping w.r.t $\alpha$. A smaller $\mathcal{L}(A)$
indicates $A$ is closer to a scaled identity mapping. Additionally, a scaled
identity mapping should have it’s weight centered on the diagonal, thus we
additionally defined $\mathcal{D}(A)$ as the average ratio of the diagonal
elements to the off-diagonal elements. A larger $\mathcal{D}(A)$ indicates the
weight matrix is more diagonal dominant.

Analysis. We first proof for a matrix $N^{n\times n}$ with weights randomly
initialized subject to $\mathcal{N}(\textbf{0},I_{n})$, we have
$\mathbb{E}[\mathcal{L}(N)]=1-\frac{1}{n^{2}}$. In our case $n=1024$, so
$\mathbb{E}[\mathcal{L}(N)]=1-\frac{1}{2^{20}}\approx 1$.

We first simply $\mathbb{E}[\mathcal{L}(N)]$:

$\displaystyle n^{2}\mathbb{E}[\mathcal{L}(N)]$
$\displaystyle=\mathbb{E}[\min_{\alpha}\sum_{i,j}(N_{i,j}-\alpha
I_{i,j})^{2}]$ $\displaystyle=\mathbb{E}[\sum_{i\neq
j}N^{2}_{i,j}+\min_{\alpha}\sum_{i}(N_{i,i}-\alpha)^{2}]$
$\displaystyle=n(n-1)\mathbb{E}[N^{2}_{0,0}]+\mathbb{E}[\min_{\alpha}\sum_{i}(N_{i,i}-\alpha)^{2}]\quad\texttt{weights
are i.i.d}$
$\displaystyle=n(n-1)(\mathrm{Var}(N_{0,0})+\mathbb{E}^{2}[N_{0,0}])+\mathbb{E}[\min_{\alpha}\sum_{i}(N_{i,i}-\alpha)^{2}]$
$\displaystyle=n(n-1)+\mathbb{E}[\sum_{i}(N_{i,i}-\frac{1}{n}\sum_{j}N_{j,j})^{2}]\quad\texttt{using
linear regression}$

For the second term, we have:

$\displaystyle\mathbb{E}[\sum_{i}(N_{i,i}-\frac{1}{n}\sum_{j}N_{j,j})^{2}]$
$\displaystyle=\sum_{i}\mathbb{E}[(N_{i,i}-\frac{1}{n}\sum_{j}N_{j,j})^{2}]$
$\displaystyle=\sum_{i}(\mathbb{E}[N^{2}_{i,i}]-\frac{2}{n}\sum_{j}\mathbb{E}[N_{i,i}N_{j,j}]+\frac{1}{n^{2}}\sum_{j}\mathbb{E}[N^{2}_{j,j}])$
$\displaystyle=\sum_{i}(1-\frac{2}{n}+\frac{1}{n^{2}}\times
n)\quad\texttt{$N_{i,i}\perp N_{j,j}(i\neq j),\mathbb{E}[N_{i,i}]=0$}$
$\displaystyle=n-1$

Thus we have:

$\mathbb{E}[\mathcal{L}(N)]=\frac{1}{n^{2}}[n(n-1)+(n-1)]=1-\frac{1}{n^{2}}\approx
1(n\to\infty)$

For $\mathcal{D}(N)=\frac{1}{n(n-1)}\sum_{i\neq j}N_{i,i}/N_{i,j}$, it’s well
known that the ratio of two i.i.d Gaussian random variables subjects to a
Cauchy distribution, which has no expectation. Thus we can not provide a
theoretical expectation for $\mathcal{D}(N)$. However, the reason for the non-
existence of expectation is a non-zero probability density at zero for a
Gaussian distribution. However, computers compute and store numbers with a
finite precision, thus the actual distribution is discrete. Thus, if we
manually set the probability of zero to zero, the expectation of such discrete
distribution will be extractable. As the discrete distribution is hard to
compute and differs when using different precision (e.g. float32, bfloat16),
we compute an empirical expectation of $\mathcal{D}(N)$ by randomly sample
$1\times 10^{6}$ i.i.d variables subject to Cauchy distribution and calculate
the average. The final expectation converges to 12.52.

###  B.2 More Base Models

To show the generalizability of our method across different LLMs, we test our
method on more base models. We use base models from the llama family, namely
LLAMA2-13B, LLAMA3-8B and LLAMA3-8B-instruct. We use the same training setup
as in Sec. 4.1 for image denoising task. The results can be seen in Fig. 12.
We observe that our method is able to process visual features across different
LLMs, with a consistent performance improvement over the baseline.
Interestingly, the performance of our method is not always positively
correlated with the size of the LLM, and the instruct version of Llama3 has a
lower performance than the non-instruct version. We leave the exploration of
this phenomenon for future work.

Figure 12: PSNR of denoising using different base models. MAE-r represents the
MAE reconstruction baseline.

###  B.3 Can a Single LLM Layer Process Low-level Features?

A recent work [31] reveals an interesting phenomenon: the Transformer layers
in a LLM are capable of facilitating various supervised visual tasks, such as
image classification and 3D point cloud classification. Specifically, [31]
implemented an approach where a frozen Transformer layer was added to a Vision
Transformer (ViT) backbone, and two linear layers were used for the transition
between visual features and the Transformer layer. We employed a similar
strategy, except that in [31], the ViT backbone and the linear layers were
trained together from scratch, whereas we only trained two linear layers. We
extract the frozen Transformer layer from LLAMA2-7B-instruct. Following [31]
we cancel the causal attention mask and positional embedding. Similar to , we
set an baseline of using a MLP with approximately same trainable parameters as
a baseline.

The numerical results can be seen in Fig 6. It is clear a frozen Transformer
layer extracted from LLM can help solving low-level vision tasks, with a
performance consistently higher than the MLP baseline. Moreover, we observe
that the performance variations across different layers of the llama model
align with the trends reported in [31]. Specifically, layer 8 exhibited the
best performance, with a gradual decline observed in deeper layers. This
finding allows us to extend the conclusions from [31]—that LLM layers can aid
in supervised vision/language tasks—to unsupervised visual tasks. This
suggests that the mechanisms by which LLM layers contribute to performance may
be broadly applicable across different types of visual processing tasks.

Figure 13: PSNR of denoising using different layers in LLM. MLP represents the
MLP baseline.

##  Appendix C Visualizations

###  C.1 Failure Case

As we’re using an auto-regressive scheme for visual feature generation,
inevitably there will be cases that auto-regressive generation falls into
wrong generation. We observe that occasionally the model will mess up the
order of visual tokens, producing misaligned images. As shown in Fig. 14, the
model fails to align the visual tokens correctly in image denoising task,
resulting in a distorted image. We note that in this case the PSNR and SSIM of
the generated image and the original image would be very low as PSNR and SSIM
apply a per-patch comparison. However, the model still denoises the images
correctly.

Figure 14: Failure case of our method. Occasionally, The model fails to align
the visual tokens correctly, resulting in a distorted image. However, the
model still denoises the images correctly. Zoom in for details.

###  C.2 Visualizations of LM4LV

We give more visualizations of LM4LV in Fig. 15.

Figure 15: Visualizations of LM4LV on various tasks. Top row: degraded images.
Middle row: MAE-r. Bottom row: LM4LV.


# Constraints on primordial black holes with CMB spectral distortions

Yupeng Yang  School of Physics and Physical Engineering, Qufu Normal
University, Qufu, Shandong, 273165, China

###### Abstract

In the mixed dark matter scenarios consisting of primordial black holes (PBHs)
and weakly interacting massive particles (WIMPs), WIMPs can be accreted onto
PBHs to form ultracompact minihalos (UCMHs) with a density spike in the early
universe. Compared with the classical dark matter halo, UCMHs are formed
earlier and have a higher density of center. Since the annihilation rate is
proportional to the squared number density of WIMPs, it is expected that WIMPs
annihilation within UCMHs is enhanced and has influences on the early
universe. Between the time of recombination and matter-radiation equality, the
energy released from WIMPs annihilation within UCMHs is injected into the
Universe resulting in CMB $y$-type distortion. We investigate these effects
and derive the upper limits on the abundance of PBHs taking advantage of the
observational results of Far Infrared Absolute Spectrophotometer (FIRAS). We
find that for the WIMPs mass range $1\leq m_{\chi}\leq 1000~{}\rm GeV$, the
upper limits on the abundance of PBHs are $5\times 10^{-3}\leq\Omega_{\rm
PBH}\leq 5\times 10^{-2}$.

##  I introduction

Although the astronomical observations and related theoretical studies suggest
that about $27\%$ of the Universe is dark matter (DM), the nature of them
remains a mystery Bertone  _et al._ (2005); Jungman  _et al._ (1996); Arbey
and Mahmoudi (2021); Aghanim  _et al._ (2020). Among the DM candidates, weakly
interacting massive particles (WIMPs) is the most competitive one Bertone  _et
al._ (2005); Jungman  _et al._ (1996). Although many experiments have been
performed in order to find WIMPs, no confirmed signal has been found so far.
Nevertheless, the properties of WIMPs can be constrained in light of the null
detection results Arcadi  _et al._ (2018); Albert  _et al._ (2017); Abdallah
_et al._ (2016); Ahnen  _et al._ (2016). Recently, the primordial black hole
(PBH) dark matter model has received renewed interest especially due to the
direct detection of gravitational waves Bird  _et al._ (2016); Deng (2021);
Franciolini  _et al._ (2022); Wang  _et al._ (2018); Abbott  _et al._ (2016).

PBHs can be formed in the early universe via the collapse of large density
perturbation Carr  _et al._ (2010); Carr (2005). The PBHs mass covers a wide
range and their abundance can be constrained using different astronomical
observations (see, e.g., Refs. Carr  _et al._ (2021a); Carr and Kuhnel (2022);
Oncins (2022) for a review, and Refs. Boshkayev  _et al._ (2022, 2022);
Kurmanov  _et al._ (2022); Boshkayev  _et al._ (2021, 2020); D’Agostino  _et
al._ (2022) for the accretion cases). By studying existing observational
results, it has been found that PBHs in most mass ranges can only constitute
part of DM Carr and Kuhnel (2022). For this case, we assume here that DM is
mainly composed of two components. One is general particle dark matter, such
as WIMPs, and the other is PBHs. In the following, we will use DM to represent
WIMPs unless otherwise specified. Both theoretical research and numerical
simulations have shown that due to the accretion of DM onto PBHs, ultracompact
minihalos (UCMHs) can be formed around PBHs with a density profile $\rho_{\rm
DM}(r)\sim\rho^{-9/4}$ Ricotti and Gould (2009); Eroshenko (2016). It is
expected that the annihilation rate of DM particles within UCMHs is very
large, because it is proportional to the square number density. The influences
of DM annihilation within UCMHs on the astronomical observations have been
investigated by previous works, while those studies have focused on the epoch
after recombination $z\lesssim z_{\rm rec}\simeq 1100$ Scott and Sivertsson
(2009); Lacki and Beacom (2010); Josan and Green (2010); Zhang (2011); Yang
_et al._ (2011); Bringmann  _et al._ (2012); Yang  _et al._ (2013, 2017); Yang
(2016); Zhang  _et al._ (2022); Cheng  _et al._ (2020); Yang (2020); Furugori
_et al._ (2020); Kadota and Tashiro (2022a); Tashiro and Kadota (2021). On the
other hand, since UCMHs can be formed before cosmological recombination, DM
annihilation within UCMHs should have influences on the early universe.

It has been pointed out that extra energy injection into the primordial plasma
in the early universe can lead to the deviation of CMB from the blackbody
spectrum McDonald  _et al._ (2001); Chluba and Sunyaev (2012); Hu and Silk
(1993); Chluba  _et al._ (2021); Bernstein and Dodelson (1990); Tashiro
(2014); Ricotti  _et al._ (2008); Sunyaev and Khatri (2013); Chluba  _et al._
(2020); Kohri  _et al._ (2014); Khlopov and Chechetkin (1987). The CMB
$y$-type and $\mu$-type distortion can be produced if extra energy is injected
in the redshift range $1100\lesssim z\lesssim 5\times 10^{4}$ and $5\times
10^{4}\lesssim z\lesssim 2\times 10^{6}$, respectively. The spectral
distortion caused by the injection of extra energy released from PBHs or DM
annihilation has been investigated in previous works McDonald  _et al._
(2001); Carr  _et al._ (2010); Pani and Loeb (2013); Nakama  _et al._ (2018a);
Tashiro and Sugiyama (2008); Chluba and Sunyaev (2012); Aalberts  _et al._
(2018); Chluba  _et al._ (2021). In this paper, we will focus on the
influences of DM annihilation within UCMHs on the CMB spectral distortion.
Since the mass of a UCMH grows slowly before the epoch of matter-radiation
equality ($z_{\rm eq}\simeq 3411$), we will focus on the redshift range
$z_{\rm rec}\leq z\leq z_{\rm eq}$. That is, we will mainly investigate the
influences of DM annihilation within UCMHs on the CMB $y$-type distortion.
Based on the observations of the Far Infrared Absolute Spectrophotometer
(FIRAS), current upper limits on the CMB $y$-type distortion is $|y|<1.5\times
10^{-5}$ Fixsen  _et al._ (1996). By requiring that the CMB spectral
distortion caused by DM annihilation within UCMHs does not exceed the existing
limits, we will derive the upper limits on the abundance of PBHs.

This paper is organized as follows. In Sec. II we investigate the basic
properties of UCMHs, and then derive the upper limits on the abundance of
PBHs. The conclusions are given in Sec. III. Throughout the paper we will use
the cosmological parameters from Planck-2018 results Aghanim  _et al._ (2020).

##  II The basic properties of UCMHs and upper limits on the abundance of PBHs

The formation mechanism of UCMHs has been studied in previous works, see,
e.g., Refs. Ricotti and Gould (2009); Eroshenko (2016); Adamek  _et al._
(2019); Boucenna  _et al._ (2018). There are two possible mechanisms: (i)
direct collapse of large density perturbation
($10^{-4}\lesssim\delta\rho/\rho\lesssim 0.3$) in the early universe, which is
not large enough to form PBHs; (ii) accretion of DM particles onto PBHs. We
will not discuss the formation mechanism of UCMHs and adopt the second
scenario for our purpose.

In the radiation dominated early universe, a large density perturbation,
$\delta\rho/\rho\gtrsim 0.3$, can directly collapse to form PBHs Carr  _et
al._ (2010); Carr (2005). Then DM particles can be accreted onto PBHs
resulting in the formation of UCMHs. The mass of a UCMH changes slowly during
the radiation dominated period and increases significantly after the redshift
of matter-radiation equality $z_{\rm eq}$ Ricotti and Gould (2009). The
specific form of the changes of mass with redshift is Ricotti and Gould
(2009); Scott and Sivertsson (2009)

$\displaystyle M_{\rm UCMH}=\left(\frac{1+z_{\rm eq}}{1+z}\right)M_{\rm PBH}$
(1)

The density profile of DM in a UCMH is in the form of Scott and Sivertsson
(2009); Zhang (2011)

$\displaystyle\rho_{\rm DM}(r)=\frac{3M_{\rm UCMH}(z)}{16\pi R_{\rm
UCMH}(z)^{3/4}r^{9/4}},$ (2)

where $R_{\rm UCMH}(z)$ is the radius of UCMH Ricotti and Gould (2009),

$\displaystyle R_{\rm UCMH}(z)=0.019~{}{\rm
pc}\left(\frac{1000}{1+z}\right)\left(\frac{M_{\rm
UCMH}(z)}{M_{\odot}}\right)^{1/3}$ (3)

For the density profile shown in Eq. (2), $\rho_{\rm DM}(r)\to\infty$ for
$r\to 0$. Taking into account the annihilation of DM particles, there is a
maximum core density, $\rho_{\rm max}$, at the center of a UCMH Scott and
Sivertsson (2009); Yang  _et al._ (2011),

$\displaystyle\rho_{\rm max}=\frac{m_{\chi}}{\left<\sigma v\right>(t-t_{i})},$
(4)

where $t_{i}$ is the formation time of UCMH and here we set $t_{i}=t_{\rm
eq}$. $m_{\chi}$ and $\left<\sigma v\right>$ are the mass and thermally
averaged annihilation cross section of DM particles, respectively. The
corresponding core radius, $r_{\rm core}$, is defined as $\rho_{\rm
max}=\rho_{\rm DM}(r_{\rm core})$. In short, the density profile of the DM
particle within a UCMH used here is as follows:

$\displaystyle\rho(r)=\left\\{\begin{array}[]{rcl}\rho_{\rm max}&&{r\leq
r_{\rm core}}\\\ \rho_{\rm DM}(r)&&{r\geq r_{\rm core}}\\\ \end{array}\right.$
(7)

The annihilation rate of the DM particle within a UCMH can be written as

$\displaystyle\Gamma_{\rm anni}$ $\displaystyle=\int_{0}^{R_{\rm UCMH}}2\pi
n^{2}_{\rm DM}(r)\left<\sigma v\right>r^{2}dr$ (8)
$\displaystyle=2\pi\frac{\left<\sigma v\right>}{m_{\chi}^{2}}\int_{0}^{R_{\rm
UCMH}}\rho_{\rm DM}^{2}(r)r^{2}dr$ $\displaystyle=2\pi\frac{\left<\sigma
v\right>}{m_{\chi}^{2}}\left[\int_{0}^{r_{\rm core}}\rho_{\rm
max}^{2}+\int_{r_{\rm core}}^{R_{\rm UCMH}}\rho_{\rm DM}^{2}\right]r^{2}dr$

In the redshift range $1100\lesssim z\lesssim 5\times 10^{4}$, the heated
electrons produced by Compton scattering or pair production on ions transfer
their energy by inverse Compton scattering to the CMB photons, producing a
distorted spectrum with phase-space distribution McDonald  _et al._ (2001);
Chluba and Sunyaev (2012); Hu and Silk (1993),

$\displaystyle f(x,y)\simeq
f(x,0)+y\frac{xe^{x}}{(e^{x}-1)^{2}}\left[\frac{x}{{\rm tanh}(x/2)}-4\right],$
(9)

where $f(x,0)=1/(e^{x}-1)$ is the Planck distribution with $x=E/T$. $y$ is the
parameter used to measure the strength of CMB $y$-type distortion. The
distribution function $f(x,y)$ is governed by the Kompaneets equation
Kompaneets (1957), and after some algebraic calculations one can find the
relation between the parameter $y$ and the energy injected into the Universe
McDonald  _et al._ (2001); Chluba and Sunyaev (2012),

$\displaystyle y=\frac{1}{4}\frac{\delta\rho_{\gamma}}{\rho_{\gamma}},$ (10)

where $\delta\rho_{\gamma}$ is the injected energy and $\rho_{\gamma}$ is the
energy density of the CMB photons.

The energy injection rate of DM annihilation per unit volume can be written as

$\displaystyle E_{\rm inj}=n_{\rm PBH}m_{\chi}\Gamma_{\rm anni}=\Omega_{\rm
PBH}\rho_{c,0}\frac{m_{\chi}}{M_{\rm PBH}}\Gamma_{\rm anni},$ (11)

where $n_{\rm PBH}=\rho_{\rm PBH}/M_{\rm PBH}$ is the number density of PBH,
$\rho_{c,0}$ is the critical density of the Universe at $z=0$ and $\Omega_{\rm
PBH}=\rho_{\rm PBH}/\rho_{c,0}$. In this paper, we have adopted a
monochromatic PBH mass function for our calculations.

Using the above equations, the parameter $y$ caused by DM annihilation within
UCMHs can be rewritten as follows:

$\displaystyle y_{\rm DM}$
$\displaystyle=\frac{1}{4}\frac{\delta\rho_{\gamma}}{\rho_{\gamma}}=\frac{1}{4}\int^{t_{2}}_{t_{1}}\frac{{\dot{\rho}_{\rm
anni}}}{\rho_{\gamma}}dt=\frac{1}{4}\int^{t_{2}}_{t_{1}}\frac{E_{\rm
inj}(z)}{\rho_{\gamma 0}(1+z)^{4}}dt$ (12)
$\displaystyle=\frac{1}{4}\frac{1}{\rho_{\gamma 0}}\int^{z_{\rm eq}}_{z_{\rm
rec}}\frac{E_{\rm inj}(z)}{H(z)(1+z)^{5}}dz,$

where $H(z)=H_{0}\sqrt{\Omega_{\rm
m}(1+z)^{3}+\Omega_{\Lambda}+{\Omega_{\gamma}}(1+z)^{4}}$, $\rho_{\gamma 0}$
is the energy density of CMB at $z=0$. Here we have used the relation
$dt=1/H(z)(1+z)~{}dz$

Using Eq. (12) and by requiring $y_{\rm DM}<1.5\times 10^{-5}$, we derive the
upper limits on the abundance of PBHs and the results are shown in Fig. 1.
From this plot, it can be found that for the DM mass range $1\leq m_{\chi}\leq
10^{3}~{}\rm GeV$ with $\left<\sigma v\right>=3\times 10^{-26}~{}\rm
cm^{3}~{}s^{-1}$, the upper limits on the abundance of PBHs are $5\times
10^{-3}\leq\Omega_{\rm PBH}\leq 5\times 10^{-2}$, and a fit to the boundary
line is as follows:

$\displaystyle\Omega_{\rm PBH}\leq 5\times 10^{-3}~{}\left(\frac{m_{\chi}}{\rm
GeV}\right)^{\frac{1}{3}}.$ (13)

Note that the limits are independent on the PBH mass, and similar features can
be also found in previous works Zhang  _et al._ (2022); Scott and Sivertsson
(2009); Yang  _et al._ (2011, 2011); Kadota and Tashiro (2022b). In fact,
since the annihilation rate of the DM particle within UCMH is proportional to
the PBH mass $\Gamma_{\rm anni}\propto M_{\rm PBH}$, the energy injection rate
per unit volume is $E_{\rm inj}\propto n_{\rm PBH}\Gamma_{\rm anni}\propto
M_{\rm PBH}^{-1}M_{\rm PBH}$. As shown in Fig. 1, the limits are stronger for
lower DM mass due to the larger number density.

Figure 1: Upper limits on the abundance of PBHs, $\Omega_{\rm PBH}=\rho_{\rm
PBH}/\rho_{c,0}$, by requiring $y_{\rm DM}<1.5\times 10^{-5}$. Here we have
set the thermally averaged annihilation cross section $\left<\sigma
v\right>=3\times 10^{-26}~{}\rm cm^{3}~{}s^{-1}$.

For comparison, the upper limits on the fraction of DM in PBHs, $f_{\rm
PBH}=\Omega_{\rm PBH}/\Omega_{\rm DM}$, from several other observations are
shown in Fig. 2 (here $\Omega_{\rm DM}=\Omega_{\rm WIMPs}+\Omega_{\rm PBHs}$).
From this plot, it can be seen that our limits provide a useful complement for
the mass range $10^{-3}\lesssim M_{\rm PBH}\lesssim 1~{}M_{\odot}$, where the
limits are obtained mainly by investigating the gravitational lensing effects
based on the results of the European Southern Observatory (EROS) Tisserand
_et al._ (2007).

Figure 2: Comparison with a few other constraints on the fraction of DM in
PBHs, $f_{\rm PBH}=\Omega_{\rm PBH}/\Omega_{\rm DM}$. The horizontal dashed
lines show the upper limits obtained by us for the DM mass $m_{\chi}=1,10,100$
and 1000 GeV. Other constraints are from: 1) the merger rate of PBHs in light
of the sensitivity of LIGO/Virgo (labeled ’LIGO/Virgo’) Vaskonen and Veermäe
(2020) (updated constraints from the O3 observational run can be found in Ref.
Hütsi  _et al._ (2021)); 2) the influence of PBHs on the dynamical evolution
of stars in the dwarf galaxy Segue I (labeled ‘Segue I’) Koushiappas and Loeb
(2017); 3) investigating the influence of accreting PBHs on the CMB with
Planck data (labeled ‘Planck’) Poulin  _et al._ (2017); 4) the gravitational
lensing effects based on EROS (labeled ‘EROS’) Tisserand  _et al._ (2007); 5)
the contribution of DM annihilation within UCMHs to the EGB (labeled ‘EGB’)
Zhang  _et al._ (2022);and 6) the influence of DM annihilation within UCMHs on
the CMB (labeled ‘CMB’) Yang  _et al._ (2011).

In the mixed dark matter scenarios investigated here the observations of
$\gamma$-ray and CMB can be also used to constrain the abundance of PBHs. For
example, the high energy photons produced by the DM annihilation within UCMHs
can contribute to the extragalactic $\gamma$-ray background (EGB) Zhang  _et
al._ (2022); Scott and Sivertsson (2009); Yang  _et al._ (2011, 2011); Yang
(2016); Boucenna  _et al._ (2018); Carr  _et al._ (2021b, c); Nakama  _et al._
(2018b); Ginés  _et al._ (2022); Belotsky  _et al._ (2005). Using the
observations of EGB by, e.g., the Fermi telescope, the upper limits on the
abundance of PBHs can be obtained Zhang  _et al._ (2022); Yang  _et al._
(2011); Yang (2016). The authors of Zhang  _et al._ (2022) used the updated
Fermi-LAT EGB measurement to derive the upper limits on the abundance of PBHs.
They found that for the $b\bar{b}$ annihilation channel the upper limit is
$f_{\rm UCMH}\sim 6\times 10^{-5}$ for $m_{\chi}=10~{}\rm GeV$ with
$\left<\sigma v\right>=3\times 10^{-26}~{}\rm cm^{3}~{}s^{-1}$ (shown in Fig.
2, horizontal dotted line labeled ‘EGB’). After recombination, the energy
released from DM annihilation within UCMHs can be injected into the
intergalactic medium (IGM), leading to the changes of the IGM thermal history
Yang  _et al._ (2011); Zhang (2011); Yang  _et al._ (2011). These changes can
be constrained by, e.g., the CMB observations. In Ref. Yang  _et al._ (2011),
the authors have used the CMB data to get the upper limits on the abundance of
PBHs. They found that the upper limit is $f_{\rm UCMH}\sim 6\times 10^{-3}$
for $m_{\chi}=10~{}\rm GeV$ with $\left<\sigma v\right>=3\times 10^{-26}~{}\rm
cm^{3}~{}s^{-1}$ (shown in in Fig. 2, horizontal dotted line labeled ‘CMB’).

##  III conclusions

We have investigated the abundance of PBHs in the mixed dark matter scenarios
consisting of PBHs and WIMPs. In this scenario, UCMHs can be formed around
PBHs with a density profile $\rho(r)\sim r^{-9/4}$. The annihilation rate of
particle DM (WIMPs) within UCMHs is enhanced and has influence on the
evolution of the Universe. Previous works have focused these influences on the
epoch after recombination. Since UCMHs can be formed in the early universe,
they can effect CMB $y$-type spectral distortion in the redshift range $z_{\rm
rec}\leq z\leq z_{\rm eq}$. We have investigated the influence of DM
annihilation within UCMHs on the CMB $y$-type distortion. By requiring that
the parameter $y$ caused by DM annihilation within UCMHs does not exceed the
current limits $|y|<1.5\times 10^{-5}$, we derived the upper limits on the
abundance of PBHs, $5\times 10^{-3}\leq\Omega_{\rm PBH}\leq 5\times 10^{-2}$
for the WIMPs mass range $1\leq m_{\chi}\leq 1000~{}\rm GeV$. Compared with
other existing limits, although our limits are not the strongest, they are new
results obtained in a different way.

##  IV Acknowledgements

This work is supported by the Shandong Provincial Natural Science Foundation
(Grant No.ZR2021MA021). Y. Yang is supported in part by the Youth Innovations
and Talents Project of Shandong Provincial Colleges and Universities (Grant
No. 201909118).

## References

  * Bertone  _et al._ (2005) G. Bertone, D. Hooper, and J. Silk, Phys. Rept. 405, 279 (2005), arXiv:hep-ph/0404175 [hep-ph] .
  * Jungman  _et al._ (1996) G. Jungman, M. Kamionkowski, and K. Griest, Phys. Rept. 267, 195 (1996), arXiv:hep-ph/9506380 [hep-ph] .
  * Arbey and Mahmoudi (2021) A. Arbey and F. Mahmoudi, Prog. Part. Nucl. Phys. 119, 103865 (2021), arXiv:2104.11488 [hep-ph] .
  * Aghanim  _et al._ (2020) N. Aghanim _et al._ (Planck), Astron. Astrophys. 641, A6 (2020), [Erratum: Astron.Astrophys. 652, C4 (2021)], arXiv:1807.06209 [astro-ph.CO] .
  * Arcadi  _et al._ (2018) G. Arcadi, M. Dutra, P. Ghosh, M. Lindner, Y. Mambrini, M. Pierre, S. Profumo, and F. S. Queiroz, Eur. Phys. J. C 78, 203 (2018), arXiv:1703.07364 [hep-ph] .
  * Albert  _et al._ (2017) A. Albert _et al._ (Fermi-LAT, DES), Astrophys. J. 834, 110 (2017), arXiv:1611.03184 [astro-ph.HE] .
  * Abdallah  _et al._ (2016) H. Abdallah _et al._ (H.E.S.S.), Phys. Rev. Lett. 117, 111301 (2016), arXiv:1607.08142 [astro-ph.HE] .
  * Ahnen  _et al._ (2016) M. L. Ahnen _et al._ (MAGIC, Fermi-LAT), JCAP 02, 039 (2016), arXiv:1601.06590 [astro-ph.HE] .
  * Bird  _et al._ (2016) S. Bird, I. Cholis, J. B. Muñoz, Y. Ali-Haïmoud, M. Kamionkowski, E. D. Kovetz, A. Raccanelli, and A. G. Riess, Phys. Rev. Lett. 116, 201301 (2016), arXiv:1603.00464 [astro-ph.CO] .
  * Deng (2021) H. Deng, JCAP 04, 058 (2021), arXiv:2101.11098 [astro-ph.CO] .
  * Franciolini  _et al._ (2022) G. Franciolini, V. Baibhav, V. De Luca, K. K. Y. Ng, K. W. K. Wong, E. Berti, P. Pani, A. Riotto, and S. Vitale, Phys. Rev. D 105, 083526 (2022), arXiv:2105.03349 [gr-qc] .
  * Wang  _et al._ (2018) S. Wang, Y.-F. Wang, Q.-G. Huang, and T. G. F. Li, Phys. Rev. Lett. 120, 191102 (2018), arXiv:1610.08725 [astro-ph.CO] .
  * Abbott  _et al._ (2016) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. Lett. 116, 061102 (2016), arXiv:1602.03837 [gr-qc] .
  * Carr  _et al._ (2010) B. J. Carr, K. Kohri, Y. Sendouda, and J. Yokoyama, Phys. Rev. D 81, 104019 (2010).
  * Carr (2005) B. J. Carr, in  _59th Yamada Conference on Inflating Horizon of Particle Astrophysics and Cosmology Tokyo, Japan, June 20-24, 2005_ (2005) arXiv:astro-ph/0511743 [astro-ph] .
  * Carr  _et al._ (2021a) B. Carr, K. Kohri, Y. Sendouda, and J. Yokoyama, Rept. Prog. Phys. 84, 116902 (2021a), arXiv:2002.12778 [astro-ph.CO] .
  * Carr and Kuhnel (2022) B. Carr and F. Kuhnel, SciPost Phys. Lect. Notes 48, 1 (2022), arXiv:2110.02821 [astro-ph.CO] .
  * Oncins (2022) M. Oncins, (2022), arXiv:2205.14722 [astro-ph.CO] .
  * Boshkayev  _et al._ (2022) K. Boshkayev, T. Konysbayev, Y. Kurmanov, O. Luongo, and D. Malafarina, (2022), arXiv:2205.04208 [gr-qc] .
  * Kurmanov  _et al._ (2022) E. Kurmanov, K. Boshkayev, R. Giambò, T. Konysbayev, O. Luongo, D. Malafarina, and H. Quevedo, Astrophys. J. 925, 210 (2022), arXiv:2110.15402 [astro-ph.HE] .
  * Boshkayev  _et al._ (2021) K. Boshkayev, T. Konysbayev, E. Kurmanov, O. Luongo, D. Malafarina, and H. Quevedo, Phys. Rev. D 104, 084009 (2021), arXiv:2106.04932 [gr-qc] .
  * Boshkayev  _et al._ (2020) K. Boshkayev, A. Idrissov, O. Luongo, and D. Malafarina, Mon. Not. Roy. Astron. Soc. 496, 1115 (2020), arXiv:2006.01269 [astro-ph.HE] .
  * D’Agostino  _et al._ (2022) R. D’Agostino, R. Giambò, and O. Luongo, (2022), arXiv:2204.02098 [gr-qc] .
  * Ricotti and Gould (2009) M. Ricotti and A. Gould, Astrophys. J. 707, 979 (2009), arXiv:0908.0735 [astro-ph.CO] .
  * Eroshenko (2016) Yu. N. Eroshenko, Astron. Lett. 42, 347 (2016), [Pisma Astron. Zh.42,no.6,359(2016)], arXiv:1607.00612 [astro-ph.HE] .
  * Scott and Sivertsson (2009) P. Scott and S. Sivertsson, Phys. Rev. Lett. 103, 211301 (2009).
  * Lacki and Beacom (2010) B. C. Lacki and J. F. Beacom, Astrophys. J. Lett. 720, L67 (2010), arXiv:1003.3466 [astro-ph.CO] .
  * Josan and Green (2010) A. S. Josan and A. M. Green, Phys. Rev. D 82, 083527 (2010), arXiv:1006.4970 [astro-ph.CO] .
  * Zhang (2011) D. Zhang, MNRAS 418, 1850 (2011), arXiv:1011.1935 [astro-ph.CO] .
  * Yang  _et al._ (2011) Y. Yang, X. Huang, X. Chen, and H. Zong, Phys. Rev. D 84, 043506 (2011), arXiv:1109.0156 [astro-ph.CO] .
  * Bringmann  _et al._ (2012) T. Bringmann, P. Scott, and Y. Akrami, Phys. Rev. D 85, 125027 (2012), arXiv:1110.2484 [astro-ph.CO] .
  * Yang  _et al._ (2013) Y. Yang, G. Yang, X. Huang, X. Chen, T. Lu, and H. Zong, Phys. Rev. D 87, 083519 (2013), arXiv:1206.3750 [astro-ph.HE] .
  * Yang  _et al._ (2017) F. Yang, M. Su, and Y. Zhao, (2017), arXiv:1712.01724 [astro-ph.HE] .
  * Yang (2016) Y. Yang, Eur. Phys. J. Plus 131, 432 (2016), arXiv:1612.06559 [astro-ph.CO] .
  * Zhang  _et al._ (2022) X.-F. Zhang, J.-G. Cheng, B.-Y. Zhu, T.-C. Liu, Y.-F. Liang, and E.-W. Liang, Phys. Rev. D 105, 043011 (2022), arXiv:2109.09575 [astro-ph.CO] .
  * Cheng  _et al._ (2020) J.-G. Cheng, S. Li, Y.-Y. Gan, Y.-F. Liang, R.-J. Lu, and E.-W. Liang, Mon. Not. Roy. Astron. Soc. 497, 2486 (2020).
  * Yang (2020) Y. Yang, Eur. Phys. J. Plus 135, 690 (2020), arXiv:2008.11859 [astro-ph.CO] .
  * Furugori  _et al._ (2020) K. Furugori, K. T. Abe, T. Tanaka, D. Hashimoto, H. Tashiro, and K. Hasegawa, Mon. Not. Roy. Astron. Soc. 494, 4334 (2020), arXiv:2002.04817 [astro-ph.CO] .
  * Kadota and Tashiro (2022a) K. Kadota and H. Tashiro, JCAP 03, 045 (2022a), arXiv:2112.04179 [astro-ph.CO] .
  * Tashiro and Kadota (2021) H. Tashiro and K. Kadota, Phys. Rev. D 103, 123532 (2021), arXiv:2104.09738 [astro-ph.CO] .
  * McDonald  _et al._ (2001) P. McDonald, R. J. Scherrer, and T. P. Walker, Phys. Rev. D 63, 023001 (2001), arXiv:astro-ph/0008134 .
  * Chluba and Sunyaev (2012) J. Chluba and R. A. Sunyaev, Mon. Not. Roy. Astron. Soc. 419, 1294 (2012), arXiv:1109.6552 [astro-ph.CO] .
  * Hu and Silk (1993) W. Hu and J. Silk, Phys. Rev. D 48, 485 (1993).
  * Chluba  _et al._ (2021) J. Chluba _et al._ , Exper. Astron. 51, 1515 (2021), arXiv:1909.01593 [astro-ph.CO] .
  * Bernstein and Dodelson (1990) J. Bernstein and S. Dodelson, Phys. Rev. D 41, 354 (1990).
  * Tashiro (2014) H. Tashiro, Progress of Theoretical and Experimental Physics 2014 (2014).
  * Ricotti  _et al._ (2008) M. Ricotti, J. P. Ostriker, and K. J. Mack, Astrophys. J. 680, 829 (2008), arXiv:0709.0524 [astro-ph] .
  * Sunyaev and Khatri (2013) R. A. Sunyaev and R. Khatri, Int. J. Mod. Phys. D 22, 1330014 (2013), arXiv:1302.6553 [astro-ph.CO] .
  * Chluba  _et al._ (2020) J. Chluba, A. Ravenni, and S. K. Acharya, Mon. Not. Roy. Astron. Soc. 498, 959 (2020), arXiv:2005.11325 [astro-ph.CO] .
  * Kohri  _et al._ (2014) K. Kohri, T. Nakama, and T. Suyama, Phys. Rev. D 90, 083514 (2014), arXiv:1405.5999 [astro-ph.CO] .
  * Khlopov and Chechetkin (1987) M. Y. Khlopov and V. M. Chechetkin, Sov. J. Particles Nucl. (Engl. Transl.); (United States) (1987).
  * Pani and Loeb (2013) P. Pani and A. Loeb, Phys. Rev. D 88, 041301 (2013), arXiv:1307.5176 [astro-ph.CO] .
  * Nakama  _et al._ (2018a) T. Nakama, B. Carr, and J. Silk, Phys. Rev. D97, 043525 (2018a), arXiv:1710.06945 [astro-ph.CO] .
  * Tashiro and Sugiyama (2008) H. Tashiro and N. Sugiyama, Phys. Rev. D78, 023004 (2008), arXiv:0801.3172 [astro-ph] .
  * Aalberts  _et al._ (2018) J. L. Aalberts, S. Ando, W. M. Borg, E. Broeils, J. Broeils, S. Broeils, B. J. Kavanagh, G. Leguijt, M. Reemst, D. R. van Arneman, and H. Vu, Phys. Rev. D 98, 023001 (2018).
  * Fixsen  _et al._ (1996) D. J. Fixsen, E. S. Cheng, J. M. Gales, J. C. Mather, R. A. Shafer, and E. L. Wright, Astrophys. J. 473, 576 (1996), arXiv:astro-ph/9605054 .
  * Adamek  _et al._ (2019) J. Adamek, C. T. Byrnes, M. Gosenca, and S. Hotchkiss, Phys. Rev. D100, 023506 (2019), arXiv:1901.08528 [astro-ph.CO] .
  * Boucenna  _et al._ (2018) S. M. Boucenna, F. Kuhnel, T. Ohlsson, and L. Visinelli, JCAP 1807, 003 (2018), arXiv:1712.06383 [hep-ph] .
  * Yang  _et al._ (2011) Y. Yang, X. Chen, T. Lu, and H. Zong, European Physical Journal Plus 126, 123 (2011), arXiv:1112.6228 [astro-ph.HE] .
  * Kompaneets (1957) A. S. Kompaneets, Soviet Journal of Experimental and Theoretical Physics 4, 730 (1957).
  * Yang  _et al._ (2011) Y. Yang, L. Feng, X. Huang, X. Chen, T. Lu, and H. Zong, Journal of Cosmology and Astroparticle Physics 2011, 020 (2011).
  * Kadota and Tashiro (2022b) K. Kadota and H. Tashiro, (2022b), arXiv:2204.13273 [hep-ph] .
  * Tisserand  _et al._ (2007) P. Tisserand _et al._ (EROS-2), Astron. Astrophys. 469, 387 (2007), arXiv:astro-ph/0607207 .
  * Vaskonen and Veermäe (2020) V. Vaskonen and H. Veermäe, Phys. Rev. D 101, 043015 (2020), arXiv:1908.09752 [astro-ph.CO] .
  * Hütsi  _et al._ (2021) G. Hütsi, M. Raidal, V. Vaskonen, and H. Veermäe, JCAP 03, 068 (2021), arXiv:2012.02786 [astro-ph.CO] .
  * Koushiappas and Loeb (2017) S. M. Koushiappas and A. Loeb, Phys. Rev. Lett. 119, 041102 (2017), arXiv:1704.01668 [astro-ph.GA] .
  * Poulin  _et al._ (2017) V. Poulin, P. D. Serpico, F. Calore, S. Clesse, and K. Kohri, Phys. Rev. D96, 083524 (2017), arXiv:1707.04206 [astro-ph.CO] .
  * Carr  _et al._ (2021b) B. Carr, F. Kuhnel, and L. Visinelli, Mon. Not. Roy. Astron. Soc. 501, 2029 (2021b), arXiv:2008.08077 [astro-ph.CO] .
  * Carr  _et al._ (2021c) B. Carr, F. Kuhnel, and L. Visinelli, Mon. Not. Roy. Astron. Soc. 506, 3648 (2021c), arXiv:2011.01930 [astro-ph.CO] .
  * Nakama  _et al._ (2018b) T. Nakama, T. Suyama, K. Kohri, and N. Hiroshima, Phys. Rev. D 97, 023539 (2018b), arXiv:1712.08820 [astro-ph.CO] .
  * Ginés  _et al._ (2022) E. U. Ginés, S. J. Witte, and O. Mena, (2022), arXiv:2207.09481 [astro-ph.CO] .
  * Belotsky  _et al._ (2005) K. M. Belotsky, M. Y. Khlopov, S. V. Legonkov, and K. I. Shibaev, Grav. Cosmol. 11, 27 (2005), arXiv:astro-ph/0504621 .


$\zeta_{\pi_{1}}\smalltriangleleft\sigma_{\pi_{1}}\wedge\dots\wedge\zeta_{\pi_{m}}\smalltriangleleft\sigma_{\pi_{m}}$.
It suffices to show that
$\emptyset[\pi_{1},\zeta_{1}]\dots[\pi_{m},\zeta_{m}]\nmodels\exists\pi_{m+1}.~{}\dots\exists\pi_{n}.~{}\psi$
as this implies that $\zeta_{1},\dots\zeta_{m}$ are a counterexample proving
that $P$ does not satifsfy $\varphi$.

Proof by contradiction. Assume that
$\emptyset[\pi_{1},\zeta_{1}]\dots[\pi_{m},\zeta_{m}]\models\exists\pi_{m+1}.~{}\dots\exists\pi_{n}.~{}\psi$.
Then, there are traces $\sigma_{\pi_{m+1}},\dots\sigma_{\pi_{n}}$ and
computations $\zeta_{\pi_{m+1}}\dots\zeta_{\pi_{n}}$ such that
$\zeta_{\pi_{m+1}}\smalltriangleleft\sigma_{\pi_{m+1}}\wedge\dots\wedge\zeta_{\pi_{n}}\smalltriangleleft\sigma_{\pi_{n}}$
and
$\hat{\zeta}^{\prime}=\emptyset[\pi_{1},\zeta_{\pi_{1}}]\dots[\pi_{n},\zeta_{\pi_{n}}]\models\psi$.
Set
$\sigma^{\prime}_{t}=({(\sigma_{\pi_{1}})_{\pi_{1}}}_{t};\dots;{(\sigma_{\pi_{n}})_{\pi_{n}}}_{t})$
and $\sigma^{\prime}=\sigma^{\prime}_{0}~{}\sigma^{\prime}_{1}\dots$. Now, by
Lemma 1 and the correctness of $A_{\psi}$, we know that
$Seq(\hat{\zeta}^{\prime})$ is accepted by $A_{\psi}$, thus
$\mathit{combine}(\sigma^{\prime},Seq(\hat{\zeta}^{\prime}))$ is accepted by
$P^{n}\otimes A_{\psi}$. Moreover, by Lemma 6, we know that
$\mathit{combine}(\sigma^{\prime},Seq(\hat{\zeta}^{\prime}))$ is also
feasible, so it is also $k$-feasible and thus accepted by $(P^{n}\otimes
A_{\psi})_{k}$. Moreover, does not end with an infeasible cycle and is thus
also accepted by $(P^{n}\otimes A_{\psi})_{k,C(k^{\prime})}$ But this means
that $\sigma$ is accepted by $(P^{n}\otimes
A_{\psi})^{\forall}_{k,C(k^{\prime})}$. and thus not by
$P^{m}\backslash(P^{n}\otimes A_{\psi})^{\forall}_{k,C(k^{\prime})}$.
Contradiction.