# Weakly-Supervised HOI Detection from Interaction Labels Only and
Language/Vision-Language Priors

Mesut Erhan Unal
Department of Computer Science
University of Pittsburgh
<EMAIL_ADDRESS>Adriana Kovashka
Department of Computer Science
University of Pittsburgh
<EMAIL_ADDRESS>
###### Abstract

Human-object interaction (HOI) detection aims to extract interacting human-
object pairs and their interaction categories from a given natural image. Even
though the labeling effort required for building HOI detection datasets is
inherently more extensive than for many other computer vision tasks, weakly-
supervised directions in this area have not been sufficiently explored due to
the difficulty of learning human-object interactions with weak supervision,
rooted in the combinatorial nature of interactions over the object and
predicate space. In this paper, we tackle HOI detection with the weakest
supervision setting in the literature, using only image-level interaction
labels, with the help of a pretrained vision-language model (VLM) and a large
language model (LLM). We first propose an approach to prune non-interacting
human and object proposals to increase the quality of positive pairs within
the bag, exploiting the grounding capability of the vision-language model.
Second, we use a large language model to query which interactions are possible
between a human and a given object category, in order to force the model not
to put emphasis on unlikely interactions. Lastly, we use an auxiliary weakly-
supervised preposition prediction task to make our model explicitly reason
about space. Extensive experiments and ablations show that all of our
contributions increase HOI detection performance.

##  1 Introduction

Figure 1: Top-left: Existing HOI detection methods need costly annotations
which contain bounding boxes for interacting human-object pairs as well as
their interaction categories. Top-right: Our method relies on image-level
interaction labels, without any information on where, between whom and how
many times those interactions occur. Bottom: During training, our approach
utilizes image captions to prune non-interacting human/object proposals with
the help of a vision-language model. Remaining human and object proposals will
be paired for classification and a large language model will verify if
predicted interactions are plausible. Best viewed in color with zoom.

Human-object interaction (HOI) detection is formally defined as correctly
localizing interacting human-object pairs and classifying their interaction in
a given natural image. The problem has been formulated in different ways,
either end-to-end, or more commonly as a two-step procedure wherein all human
and object instances get detected first, then interacting human-object pairs
are identified. Regardless of its formulation, researchers rely on strong
supervision to tackle HOI detection. This strong supervision is in the form of
bounding box annotations for interacting human-object pairs as well as
semantic labels for their interactions, which are costly to acquire and
cognitively demanding for annotators as they require one to fully understand
the image content111http://www-personal.umich.edu/~ywchao/hico/hoi-det-
ui/demo_20171121.html. Despite the excessive cost of gathering annotations for
HOI detection, weakly-supervised directions to relax this strong supervision
need have not been fully explored, due to the combinatorial complexity of
object interactions over object and predicate space.

In this paper, we tackle weakly-supervised HOI detection using the weakest
supervision in the literature, namely image-level interaction labels (e.g.
ride). This supervision level is less costly and more natural to acquire than
ones required in existing efforts, as annotators would be required to answer a
simple question: “What are the individuals doing in this picture?”. To make
learning possible, our approach utilizes free-form captions paired with images
to weakly-supervise an auxiliary task and to prune non-interacting humans and
objects. We query a large language model (LLM) to eliminate unlikely human-
object interactions (e.g. riding toothbrush). To increase the spatial
reasoning capability of our model, we further formulate an auxiliary
preposition prediction task. In this task, our model learns to assign one of
the predefined prepositions to each human-object pair during training via weak
supervision. Having free-form captions in hand also gives us the ability of
extracting image-level interaction labels using a language parser, and hence
further relax the level of supervision. Our code will be released upon
publication. To summarize, our main contributions are as follows:

  * •

We formulate a weakly-supervised HOI detection setting where supervision comes
from image-level interaction labels (e.g. ride, eat). This weak supervision
has not previously been used in the literature.

  * •

We utilize free-form captions paired with images to exploit the implicit
grounding capability of a vision language model (VLM) in order to prune non-
interacting human and object proposals.

  * •

We make use of an large language model (LLM) to verify if a given
$<$interaction, object$>$ pair is plausible.

  * •

To further increase our model’s spatial reasoning capability, we formulate a
weakly-supervised preposition prediction task.

  * •

For the first time in the literature, we train an HOI detection model using
image-caption pairs which are abundant on the web.

##  2 Related Work

###  2.1 Human-object interaction detection

The problem of detecting interactions between humans and objects was
originally introduced in [11] and has drawn immense attention in the computer
vision community since then. Most of the research efforts on this topic [9,
23, 38, 20, 22, 41] use a two-stage solution in which human/object locations
are extracted along with their semantic labels by an off-the-shelf object
detector first, and an interaction classification model is learnt on pairwise
human-object features. Apart from human/object appearances, there exist models
that make use of contextual features [9, 38], spatial layouts [23, 38, 41] and
human pose estimations [23]. Inspired by one-stage object detection efforts,
researchers lately try to formulate end-to-end HOI detection approaches where
human/instance localization and interaction classification are performed in
parallel [24, 17, 18, 19]. These methods are analogous to CNN-based (e.g.
YOLO[33]) and Transformer-based (e.g. DETR[3]) end-to-end object detectors.
PPDM[24] takes a step forward and drops the need for heuristically created
“anchors”, formulating HOI detection as a point matching problem between human
and object locations.

Regardless of being one-stage or two-stage, these methods rely on strong
supervision which is costly to acquire. This supervision is in the form of
quadruplets that contain interacting human-object locations, object category
and interaction category. Even though HOI detection is extremely costly to
supervise, there exists a lack of weakly-supervised efforts in the literature.
Among the existing weakly-supervised methods, MX-HOI [21] proposes a momentum-
independent learning framework where they utilize both weak and strong
supervision. Additionally, AlignFormer [16] formulates an alignment layer in
transformer framework, that generates pseudo-aligned human-object pairs from
weak annotations, conditioning on geometric and visual priors. Both of these
methods utilize image-level $<$interaction, object$>$ annotations (e.g. {eat-
banana}) as weak supervision as opposed to the much weaker supervision we use
in our work, namely image-level interaction labels (e.g. {eat}).

###  2.2 Using cues from vision-language models

Following vision-language models’ breakthrough, researchers have explored
their usage in aiding diverse computer vision tasks. For example, one of the
most popular VLMs, namely CLIP [32], has been researched extensively in the
context of image generation [31], cross-modal retrieval [1, 8], image
classification [1], object detection [10], HOI detection [7, 25] and image
captioning [4], thanks to its robust image-text joint space learned on a
massive dataset.

Even though [7, 25] also utilize CLIP in the context of HOI detection, how
CLIP is employed within our approach is quite different. [7] uses CLIP’s text
encoder to initialize context-aware HOI queries within a fully-supervised
Transformer-based HOI detector. [25] utilize CLIP as a teacher within their
model and distill knowledge for both visual and textual understanding of
interactions. The most similar work to ours in terms of how CLIP is employed
is ProposalCLIP [36], where authors prune low-quality object proposals
produced by a static algorithm (e.g. EdgeBoxes [42]). Their method runs
cropped proposal regions along with produced captions for object categories
(i.e. $\\{\text{``a photo of a
}c_{i}^{(obj)}\text{''}\\}_{i=1}^{|C^{(obj)}|}$) through CLIP and removes
proposals based on the alignment entropy over caption set. In our work, on the
other hand, we need to quantify if a given proposal is a part of an
interaction or not. Unlike running CLIP on large number of proposals, we run
whole image through CLIP once and create grounding maps to calculate an
interaction score on each proposal.

Figure 2: Overview of our method during training. Retrieving human and object
proposals from an object detector, our method first prunes non-interacting
human/object proposals with the help of a vision-language model, calculating
an interaction score for each proposal. Next, we pair remaining human-object
proposals and run those pairs through a two-stream feed-forward neural net
($F_{2}$) that operates on $F_{1}$’s output space. Finally, image-level
predictions are calculated by summing $F_{2}$’s output over region pairs. We
query a large-language model to restrict our model’s output space only to
meaningful interactions. In order to improve our model’s spatial reasoning
capability, we formulate a weakly-supervised preposition prediction task
wherein supervision comes from preposition extracted from captions. During
inference, we drop proposal pruning and preposition prediction modules,
requiring only an image to detect HOI instances.

##  3 Method

###  3.1 Formulating HOI detection with weak supervision

Assume an object detector that outputs a set of human and object predictions
for a given image, $\mathcal{H}=\\{h_{i}\\}_{i=1}^{N}$ and
$\mathcal{O}=\\{o_{j}\\}_{j=1}^{M}$ respectively. Each of these predictions is
in the form of $\\{x^{(1)},y^{(1)},x^{(2)},y^{(2)},c^{o},s^{o}\\}$ where
$x^{(1)},y^{(1)}$ and $x^{(2)},y^{(2)}$ denote the top-left and bottom-right
corner coordinates of the proposal bounding box respectively, $c^{o}$ is the
semantic category assigned by the object detector (“person” for each proposal
in $\mathcal{H}$) and $s^{o}\in[0,1]$ is the confidence score.

Given (1) the above set of human and object proposals, and (2) provided image-
level interaction labels (e.g. ride), our goal is to learn an HOI detection
model, $F(\cdot)$, that can map each human-object pair
$\\{h,o\\}\in\mathcal{H}\times\mathcal{O}$ to an interaction class $c^{v}$
that belongs to a predefined set of classes $C^{v}=\\{c^{v,(k)}\\}_{k=1}^{K}$
and a confidence score $s^{v}\in[0,1]$, yielding
$\mathcal{H}\times\mathcal{O}\xrightarrow{F(\cdot)}\\{c^{v}_{i,j},s^{v}_{i,j}\\}_{i=1,j=1}^{N,M}$.
Here the $v$ superscript denotes “verb” (interaction), while $k$ denotes the
specific verb/interaction class. Please note that this definition of weakly-
supervised HOI detection is slightly different than the one given in §Sec. 1
as we offload localization and semantic labeling of humans and objects to an
object detector as in every two-stage HOI detection work. We can further
rewrite $F$ as composition of two separate functions, $F_{1}$ and $F_{2}$,
where $F_{1}$ is responsible for extracting pairwise features and modeling
interactions while $F_{2}$ performs classification, yielding $F=F_{2}\circ
F_{1}$.

In the fully-supervised case, learning is facilitated by giving the model
access to the correct HOI targets
$Y=\\{\text{bbox}^{\text{human}}_{i},\text{bbox}^{\text{object}}_{i},c^{o}_{i},c^{v}_{i}\\}_{i=1}^{L}$
which contain ground-truth human and object locations as bounding box
coordinates, as well as the semantic categories for the object and the
interaction. When one has ground-truth targets at hand, an HOI detection model
can be trained to increase the likelihood of $\\{h,o\\}$ pairs that spatially
and semantically overlap with a HOI target to have the same interaction class
as that target. In our case, however, the model can only access a set of
ground-truth interaction classes for a given image, without even knowing if a
certain interaction happens once or multiple times in the image (see Figure
1).

Inspired by existing weakly-supervised object detection (WSOD) literature, we
formulate weakly-supervised HOI detection as a multiple instance learning
(MIL) problem. In this formulation, each image is considered as a bag of
human-object pairs (i.e. $\mathcal{H}\times\mathcal{O}$). If a bag (i.e.
image) is labeled positive for a certain interaction, it has to contain at
least one $\\{h,o\\}$ pair of that interaction. Similar to WSDDN [2], we split
the final classification layer, $F_{2}$, into a two-stream head (i.e.
$F_{2}^{(1)}$ and $F_{2}^{(2)}$) where one models “what is the most probable
interaction class for a given human-object pair?” (i.e.
$P(C^{v}\,|\,\\{h,o\\},\,F_{2}^{(1)})$) while the other models “what is the
most probable human-object pair for a given interaction class?” (i.e.
$P(\\{h_{i},o_{j}\\}_{i=1,j=1}^{N,M}\,|\,c^{v},\,F_{2}^{(2)})$). Assuming we
get $d$-dimensional feature for each pair through $F_{1}$ as row vectors in a
feature matrix $\mathbf{Z}$, such as
$F_{1}(\mathcal{H}\times\mathcal{O})=\mathbf{Z}\in\mathbb{R}^{NM\times d}$,
the aforementioned probabilities can be calculated by mapping $\mathbf{Z}$ to
a $|C^{v}|$-dimensional space first and then applying softmax on different
dimensions. Hence, we can formulate $F_{2}^{(1)}$ as a mapping
$\mathbb{R}^{d}\rightarrow\mathbb{R}^{|C^{v}|}$ followed by a row-wise softmax
on $\mathbf{Z}$ while $F_{2}^{(2)}$ can be formulated as a mapping
$\mathbb{R}^{d}\rightarrow\mathbb{R}^{|C^{v}|}$ followed by a column-wise
softmax on $\mathbf{Z}$. Then, we can write $F_{2}$’s output
$\mathbf{Z}^{\text{HOI}}$ as:

$\begin{split}\mathbf{Z}^{\text{HOI}}&=F_{2}^{(1)}(\mathbf{Z})\odot
F_{2}^{(2)}(\mathbf{Z})\in\mathbb{R}^{NM\times|C^{v}|}\qquad\text{where}\\\
&F_{2}^{(1)}(\mathbf{Z})=\sigma^{\rightarrow}(\mathbf{Z}*W_{F_{2}^{(1)}})\\\
&F_{2}^{(2)}(\mathbf{Z})=\sigma^{\downarrow}(\mathbf{Z}*W_{F_{2}^{(2)}})\end{split}$
(1)

where $\odot$ represents Hadamard product, $N$ is the number of human
proposals, $M$ is the number of object proposals, $|C^{v}|$ is the number of
interaction classes,
$W_{F_{2}^{(1)}},W_{F_{2}^{(2)}}\in\mathbb{R}^{d\times|C^{v}|}$ are weight
matrices, and $\sigma^{\rightarrow}$ and $\sigma^{\downarrow}$ are row-wise
and column-wise softmax operations, respectively.

Finally, we can formulate our learning objective as minimizing $|C^{v}|$
binary classification losses, one for each interaction class $c^{v}\in C^{v}$:

$\begin{split}\mathcal{L}^{\text{HOI}}(\hat{Y}^{v},Y^{v})&=\frac{1}{|C^{v}|}\sum_{k=1}^{|C^{v}|}\ell(\hat{y}^{v,(k)},y^{v,(k)})\\\
\hat{Y}^{v}&=\sum^{NM}\mathbf{Z}^{\text{HOI}}\end{split}$ (2)

where $Y^{v}$ is the binary image-level interaction labels and $y^{v,(k)}=1$
iff $c^{v,(k)}$ is apparent in the image, $0$ otherwise. Let
$\mathbf{Z}^{\text{HOI}}_{[i\times j]}$ denote the $|C^{v}|$-dimensional row
vector in $\mathbf{Z}^{\text{HOI}}$ that corresponds to the pair
$\\{h_{i},o_{j}\\}$. Then one can obtain the pair’s interaction class
$c^{v}_{i,j}$ and confidence score $s^{v}_{i,j}$ during inference as:

$\begin{split}c^{v}_{i,j}&=c^{v,(\operatorname*{arg\,max}_{k}\mathbf{Z}^{\text{HOI},(k)}_{[i\times
j]})}\\\ s^{v}_{i,j}&=\max(\mathbf{Z}^{\text{HOI}}_{[i\times j]})\cdot
s_{h_{i}}^{o}\cdot s_{o_{j}}^{o}\end{split}$ (3)

where $s_{h_{i}}^{o}$ and $s_{o_{j}}^{o}$ are confidence scores assigned to
$h_{i}$ and $o_{j}$ by the object detector, respectively.

###  3.2 Extracting interaction labels from captions

Our weakly-supervised HOI detector learning procedure requires image-level
interaction labels for supervision. However, one can utilize captions to
extract those annotations to further relax the level of annotation required.
In this work, we demonstrate how one can train an HOI detector on a dataset
scraped from the web and contains noisy captions, using a simple technique.

We start with extracting nouns and verbs from captions using a POS tagger
[14]. Consider we have a set of predefined interaction categories $C^{v}$, and
verb and noun sets for a particular image $\mathcal{V}=\\{v_{i}\\}_{i=1}^{A}$
and $\mathcal{N}=\\{n_{i}\\}_{i=1}^{B}$, respectively. For each image where
“person” $\in\mathcal{N}$, we construct its label as
$Y^{v}=\\{v\,|\,v\in\mathcal{V}\,\text{and}\,v\in C^{v}\\}$. We use a synonym
list to match “person”, as given in [30].

###  3.3 Pruning non-interacting proposals

Learning an HOI detector using only image-level interaction labels is
inherently a difficult task. A model needs to learn how to identify
interacting human-object pairs among a large candidate pool and classify their
interactions correctly. Without bounding box supervision, the model is left by
itself to learn how an interacting human or object should look like, and what
combination of those maps to a certain interaction class. For instance,
consider learning the interaction “kick” with object “ball”. We can expect
that most of the images containing this particular interaction and object
would portray a game field where more than one person is apparent. To a
weakly-supervised model, each person would be equally likely to be the subject
of the “kick” interaction. One can try to build coarse heuristic rules (e.g.
interacting human-object pairs should be close in space) or more fine-grained
ones (e.g. human-object pairs for kick interaction should be close in space,
but they can be further for another interaction) to reduce the search space
but it is impossible to precisely develop rules for every natural interaction.
To this end, we propose to exploit the implicit grounding capability of a
vision-language model to prune non-interacting human and object proposals. In
this work, we employ CLIP [32] and produce visual grounding maps for image-
text pairs using [6].

Consider that we have access to free-form captions for the images in our
training data; we do _not_ require captions at inference time. Given an image
and its caption, we first extract all verbs
$\mathcal{V}=\\{v_{i}\\}_{i=1}^{A}$ and nouns
$\mathcal{N}=\\{n_{i}\\}_{i=1}^{B}$ out of the caption using a POS tagger
[14]. We then create human captions as $HC=\\{\text{``a person is
$v_{i}$-ing''}\\}_{i=1}^{A}$ and object captions as $OC=\\{\text{``a $n_{i}$
is being $v_{j}$-ed''}\\}_{i=1,j=1}^{B,A}$. We run the image and created
captions through CLIP to produce a grounding map per caption and resize them
into original image dimensions via bilinear interpolation. Finally, grounding
maps are min-max normalized to map their values into $[0,1]$ range.

Retrieving grounding maps $GH$ and $GO$ for human and object captions
respectively, one can calculate a grounding score, $g$, for each human and
object proposal, $h\in\mathcal{H},o\in\mathcal{O}$. Intuitively, $g$ should
measure how likely a certain proposal is to engage in an interaction. We
calculate $g$ for each proposal as follows:

$\small\begin{split}g_{h}&=\frac{1}{(x^{(2)}_{h}-x^{(1)}_{h})(y^{(2)}_{h}-y^{(1)}_{h})}\frac{1}{|GH|}\sum_{k=1}^{|GH|}\sum_{i=x^{(1)}_{h},j=y^{(1)}_{h}}^{x^{(2)}_{h},y^{(2)}_{h}}GH_{i,j}^{(k)}\\\
g_{o}&=\frac{1}{(x^{(2)}_{o}-x^{(1)}_{o})(y^{(2)}_{o}-y^{(1)}_{o})}\frac{1}{|GO|}\sum_{k=1}^{|GO|}\sum_{i=x^{(1)}_{o},j=y^{(1)}_{o}}^{x^{(2)}_{o},y^{(2)}_{o}}GO_{i,j}^{(k)}\end{split}$
(4)

The above equations simply calculate the average grounding score that falls
into each human/object proposal region using the corresponding grounding maps.
Finally, the interaction score for a human proposal $h_{i}$ or an object
proposal $o_{j}$ is calculated as the multiplication of its grounding score
$g$ and confidence score given by the object detector $s^{o}$:

$I_{h_{i}}=g_{h_{i}}\cdot s_{h_{i}}^{o}\,,\,I_{o_{j}}=g_{o_{j}}\cdot
s_{o_{j}}^{o}$ (5)

The reason behind multiplying $g$ and $s^{o}$ for interaction score
calculation is quite simple. In our experiments, we have seen that the
generated grounding maps usually focus on the most distinct parts of the
interacting human/object, which would result in proposals covering only those
distinct areas to get the highest interaction scores if only $g$ was used.

Lastly, we sort human/object proposals in descending order of their
interaction scores $I_{h}$/$I_{o}$ and keep top $50\%$ as is while assigning a
special “background” class to others. These “background” proposals still get
paired with human proposals within the model and will serve as negatives.

###  3.4 Suppressing implausible interactions

Previous work [12, 41] has shown that it can be beneficial to restrict a
model’s output space only to meaningful interactions, conditioning on some
type of lookup table in which plausible interactions are encoded. While [12]
proposed to learn these conditions within the model optimizing an indicator
function over possible interactions given human and object proposals, [41]
compute them directly on data, iterating over ground-truth HOI targets. There
also exist works that learn such conditions by modeling interactions as
phrases (e.g. “person eat banana”) in a textual [29] or multi-modal space
[40]. Unlike these methods, ours does not require subject-predicate-object
annotations nor multimodal training.

In this work, we propose to use a large language model (LLM) to query which
interactions are plausible for a given object category. Our hypothesis is that
these models would have learnt natural co-occurrences throughout their
training on massive text, and this information would also be applicable to the
visual domain. We consider two natural approaches for how an LLM can be used
for this purpose: (1) inputting “A person is [MASK] $c^{o}$” caption to the
model (where $c^{o}$ denotes a particular object category) and calculating a
probability distribution over possible interaction categories
$C^{v}=\\{c^{v,(k)}\\}_{k=1}^{K}$ at the masked-language modeling (MLM) head
to obtain the [MASK] token, ignoring the rest of the vocabulary, and (2)
plugging “What a person do with $c^{o}$?” as a question and interaction
classes $C^{v}=\\{c^{v,(k)}\\}_{k=1}^{K}$ as an answer set, then retrieving
the language model’s output distribution over $C^{v}$ at the multiple choice
question answering (MCQA) head. After obtaining a probability distribution
over interaction classes given an object category i.e. $P(C^{v}\,|\,c^{o}$),
we create a binary lookup table for each object category, wherein interaction
categories are encoded as plausible (if their probability is larger than
average) or otherwise implausible.

$\begin{split}\Phi_{c^{o}}&=\\{\phi(c^{o},c^{v,(k)})\\}_{k=1}^{|C^{v}|}\\\
\phi(c^{o},c^{v,(k)})&=\left\\{\begin{array}[]{ll}1&P(c^{v,(k)}\,|\,c^{o})>\frac{1}{|C^{v}|}\sum^{|C^{v}|}P(c^{v}\,|\,c^{o})\\\
0&\text{otherwise}\\\ \end{array}\right.\end{split}$ (6)

Lastly, we double the confidence score of a human-object pair
$\\{h_{i},o_{j}\\}$ if its predicted label is plausible given the object
category of $o_{j}$:

$s^{v^{\prime}}_{i,j}=s^{v}_{i,j}\cdot(1+\phi(c_{o_{j}}^{o},c_{i,j}^{v}))$ (7)

We use RoBERTa [28] to query if a given $<$interaction, object$>$ pair is
plausible. We build $\Phi_{c^{o}}$ for each dataset before training, instead
of querying LLM constantly.

###  3.5 Formulating weakly-supervised preposition prediction

Prior work [15] demonstrates that encoding pairwise spatial relations as
discrete labels (e.g. inside of, contains) within a model improves performance
on tasks that require explicit spatial understanding, such as TextVQA [37].
Inspired from but different from their work, we formulate a preposition
prediction task in which the model is forced to learn a mapping from pairwise
features to discrete spatial labels in weakly-supervised manner, in the unique
context of human-object interaction.

Similar to our weakly-supervised HOI detection formulation given in §Sec. 3.1,
we employ a two-stream head, $F_{3}$, that operates on $F_{1}$’s output space.
Assuming our pre-defined preposition set is $C^{p}=\\{c^{p,(k)}\\}_{k=1}^{K}$
and we get a $d$-dimensional feature for each human-object pair through
$F_{1}$ as row vectors in feature matrix Z, such as
$F_{1}(\mathcal{H}\times\mathcal{O})=\mathbf{Z}\in\mathbb{R}^{NM\times d}$, we
formulate $F_{3}^{(1)}$ as a mapping
$\mathbb{R}^{d}\rightarrow\mathbb{R}^{|C^{p}|}$ followed by a row-wise softmax
on $\mathbf{Z}$ while $F_{3}^{(2)}$ is formulated as a mapping
$\mathbb{R}^{d}\rightarrow\mathbb{R}^{|C^{p}|}$ followed by a column-wise
softmax on $\mathbf{Z}$. Then, we can write $F_{3}$’s output
$\mathbf{Z}^{\text{PREP}}$ as:

$\begin{split}\mathbf{Z}^{\text{PREP}}&=F_{3}^{(1)}(\mathbf{Z})\odot
F_{3}^{(2)}(\mathbf{Z})\in\mathbb{R}^{NM\times|C^{p}|}\qquad\text{where}\\\
&F_{3}^{(1)}(\mathbf{Z})=\sigma^{\rightarrow}(\mathbf{Z}*W_{F_{3}^{(1)}})\\\
&F_{3}^{(2)}(\mathbf{Z})=\sigma^{\downarrow}(\mathbf{Z}*W_{F_{3}^{(2)}})\end{split}$
(8)

where $\odot$ represents Hadamard product, $N$ is the number of human
proposals, $M$ is the number of object proposals, $|C^{p}|$ is the number of
preposition classes,
$W_{F_{3}^{(1)}},W_{F_{3}^{(2)}}\in\mathbb{R}^{d\times|C^{p}|}$ are weight
matrices, and $\sigma^{\rightarrow}$ and $\sigma^{\downarrow}$ are row-wise
and column-wise softmax operations, respectively.

Finally, we formulate our learning objective as minimizing $|C^{p}|$ binary
classification losses, one for each preposition class $c^{p}\in C^{p}$:

$\begin{split}\mathcal{L}^{\text{PREP}}(\hat{Y}^{p},Y^{p})&=\frac{1}{|C^{p}|}\sum_{k=1}^{|C^{p}|}\ell(\hat{y}^{p,(k)},y^{p,(k)})\\\
\hat{Y}^{p}&=\sum^{NM}\mathbf{Z}^{\text{PREP}}\end{split}$ (9)

where $Y^{p}$ is the binary image-level preposition labels and $y^{p,(k)}=1$
iff $c^{p,(k)}$ is apparent in the image, $0$ otherwise. Adding a new task in
the model, our overall training objective now becomes minimizing both
$\mathcal{L}^{\text{HOI}}$ (Eq. 2) and $\mathcal{L}^{\text{PREP}}$:

$\mathcal{L}=\mathcal{L}^{\text{HOI}}+\lambda\mathcal{L}^{\text{PREP}}$ (10)

Since none of the datasets we use in our experiments comes with such
preposition annotations, we utilize captions to extract them. Specifically, we
run captions through a scene graph parser (e.g. Stanford Scene Graph Parser
[35]) to extract $<$subject, predicate, object$>$ triplets. We filter out
triplets whose subject is not “person” or predicate is not in $C^{p}$, which
we curated by hand collecting $32$ most common prepositions. We use the same
synonym list for “person”, mentioned in §Sec. 3.2. After the filtering
process, the unique predicates from the remaining triplets are used as image-
level preposition labels for their corresponding images.

##  4 Experiments

###  4.1 Setup

Datasets and metrics. We use the well-established HOI detection benchmark
datasets, HICO-DET [5] and V-COCO [11], in our experiments. HICO-DET contains
$37,633$ training and $9,546$ test images with bounding box annotations for
interacting human-object pairs and their interaction labels. There are $80$
object (same as in MS COCO [27]) and $117$ interaction categories in HICO-DET
with $600$ unique $<$interaction, object$>$ pairs. As HICO-DET instances do
not come with a paired caption, we use a state-of-the-art image captioning
model, OFA [39], to generate one for each image in the training split. On the
other hand, V-COCO is a relatively smaller dataset with $5,400$ images in
trainval and $4,946$ images in test split. There are $80$ object and $26$
interaction categories. As V-COCO is a subset of MS COCO, each image is paired
with $5$ captions. We use standard metrics for each dataset which are Agent AP
and Role AP for V-COCO, and Full mAP for HICO-DET. Please note that HICO-DET’s
Full mAP is analogous to V-COCO’s Role AP, which requires predicted human and
object bounding boxes to have at least $0.5$ IoU with corresponding HOI
target, and predicted interaction category should be the same as the target
interaction label. V-COCO’s Agent AP, on the other hand, requires correct
localization of humans (i.e. $\text{IoU}>0.5$) engaging in a particular
interaction.

Furthermore, for the first time in the literature we learn an HOI detection
model on a small subset of Conceptual Captions, which consists of roughly
$18,000$ image-caption pairs, without any HOI-related annotation. We extract
image-level interaction labels from captions as explained in §Sec. 3.2. We use
the V-COCO test split to evaluate models trained on this new dataset.

Baseline and training procedure. Please note that our proposed approach as a
whole is applicable to any existing two-stage HOI detection method. In our
experiments, we use SCG [41] as our baseline and implement our main
contributions on top of it. We choose SCG because it is one of the best
performing fully-supervised two-stage HOI detector with a publicly-available
implementation. Unless noted otherwise, we use the same hyperparameter
settings as [41]. We use Faster R-CNN[34] with ResNet50-FPN[13] pretrained on
MS COCO to generate detections. We train all models on $4\times$ NVIDIA Quadro
RTX 5000 GPUs with an initial learning rate of $1e-4$ and batch size of $16$
($4$ images per GPU). On V-COCO and HICO-DET, all models are trained for $8$
epochs, reducing learning rate to $1e-5$ after $6$th epoch. On the other hand,
we train models on Conceptual Captions subset for $5$ epochs, without applying
any decay strategy on learning rate. For weakly-supervised HOI detection task,
we use binary adaptation of focal loss [26] ($\ell$ in Eq. 2) following the
baseline, and binary cross entropy loss for weakly-supervised preposition
prediction task ($\ell$ in Eq. 9). We set the weight for weakly-supervised
preposition prediction task as $0.1$ ($\lambda$ in Eq. 10). Interested readers
may consult the original paper for additional details on model implementation
and training procedure.

Method | Sup. |  Backbone | Role AP
---|---|---|---
iCAN [9] | Full | RN50 | 52.04
VSGNet [38] | Full | RN152 | 57.00
SCG [41] | Full | RN50 FPN | 58.02
IDN [22] | Full | RN50 | 60.30
HOTR [18] | Full | RN50+Transformer | 64.40
MSTR [19] | Full | RN50+Transformer | 65.20
MX-HOI [21] | Weak+ | RN101 | -
AlignFormer [16] | Weak+ | RN50 | 14.15
Baseline [41] (§3.1) | Weak | RN50 FPN | 20.05
Ours | Weak | RN50 FPN | 29.59
Ours-CC | Weak- | RN50 FPN | 17.71
Table 1: V-COCO test Role AP performance among methods trained on V-COCO
trainval split (except OursCC). Ours outperforms AlignFormer by a large margin
(absolute $15.54\%$), even though its supervision comes from image-level
$<$interaction, object$>$ labels (Weak+) rather than image-level
$<$interaction$>$ only labels (Weak) we use. It also greatly improves
(absolute $9.54\%$) over Baseline, which is close to SOTA when trained fully-
supervised, verifying effectiveness of our contributions. Trained on a dataset
scraped from the web, extracting image-level $<$interaction$>$ only labels
from captions (Weak-), our method (OursCC) still outperforms AlignFormer by
absolute $3.56\%$. RN denotes ResNet. MX-HOI did not report V-COCO results.
Bolding shows the best method within each supervision level.

###  4.2 Comparison with the SOTA

In this subsection, we compare our model against the state-of-the-art HOI
detection efforts. We also include fully-supervised approaches to inform
readers on the performance gap between fully- and weakly-supervised HOI
detection literature, and to show that our baseline SCG [41] is comparable
with SOTA when trained with strong supervision. We would like to stress that
there are not many weakly-supervised work in the HOI detection literature and
existing approaches (e.g. MX-HOI [21] and AlignFormer [16]) use image-level
$<$interaction, object$>$ annotations as weak supervision. Careful readers
would have already noticed that this specific definition of “weak supervision”
is considerably stronger than is in our formulation as it reduces the search
space on object proposals to match with a specific interaction category in a
given image. Consider a natural image that portrays a person riding a bike
while another riding a motorcycle in an urban setting. For that particular
image, image-level $<$interaction, object$>$ annotation will yield to {ride-
bike, ride-motorcycle} while image-level interaction annotation (our
supervision) will be {ride} only. If $<$interaction, object$>$ annotation are
exploited, object proposals that can be matched with “ride” will be narrowed
down to bike and motorcycle. In our case, however, the object space is
excessively larger, including other objects that may be apparent in the image
(e.g. bus, car, etc.).

In Table 1, we compare HOI detection performance on V-COCO test split among
the models trained on the V-COCO trainval set (except Ours-CC). Our method
improves absolute $9.54\%$ over weakly-supervised variant of SCG, which we
build our contributions upon, and absolute $15.54\%$ over AlignFormer, which
uses stronger supervision in the form of image-level $<$interaction, object$>$
labels. Our method trained on the Conceptual Captions subset (Ours-CC) also
surpasses AlignFormer and achieves a comparable performance to weakly-
supervised SCG, even though we extract image-level interaction labels from
captions to supervise its learning (§Sec. 3.2) and use MS COCO trained object
detector to produce human/object proposals.

Method | Sup. |  Backbone | mAP
---|---|---|---
iCAN [9] | Full | RN50 | 14.84
VSGNet [38] | Full | RN152 | 19.80
SCG [41] | Full | RN50 FPN | 21.85
IDN [22] | Full | RN50 | 23.36
HOTR [18] | Full | RN50+Transformer | 23.46
MSTR [19] † | Full | RN50+Transformer | 31.17
MX-HOI [21] † | Weak+ | RN101 | 16.14
AlignFormer [16] † | Weak+ | RN50 | 19.26
Baseline [41] (§3.1) | Weak | RN50 FPN | 7.05
Ours | Weak | RN50 FPN | 8.38
Table 2: Full mAP (Default) comparison on HICO-DET. Unsurprisingly,
AlignFormer and MX-HOI benefit from having much stronger supervision, namely
image-level $<$interaction, object$>$ labels (Weak+), when combinatorial
complexity over interaction space is increased moving from V-COCO to HICO-DET.
However, Ours still improves over Baseline, which uses the same image-level
$<$interaction$>$ only labels (Weak) verifying effectiveness of our
contributions. RN denotes ResNet. † denotes using an object detector fine-
tuned on HICO-DET. Figure 3: Qualitative examples sampled from V-COCO test
split. Black and white boxes show interacting humans and objects,
respectively. Our method successfully detects more interactions than Baseline,
especially when the same human is subject to more than one interactions.
Moreover, the 3rd example shows that it selects the better object proposal for
“sit” interaction (horse). Interaction label explanations can be found in
original V-COCO paper [11], Table 1.

Similarly in Table 2, we compare HOI detection performance on the HICO-DET
test split among the models trained on the HICO-DET training set. Results show
that both weakly-supervised SCG and Ours fail to sustain their improvement
over AlignFormer, due to the increased number of interaction categories over
V-COCO (26 vs 117). Unsurprisingly, AlignFormer was not affected heavily by
increased combinatorial complexity over the $<$interaction, object$>$ joint
space thanks to its stronger supervision than ours. It is also worth noting
that both AlignFormer and MX-HOI use an object detector fine-tuned on HICO-DET
(denoted by ${\dagger}$) while we do not. Regardless of its unsustained
performance on HICO-DET, our method still improves over the baseline weakly-
supervised SCG by absolute $1.33\%$ (relative $18.87\%$), which has been
trained with the same level of supervision as Ours.

###  4.3 Ablation study

To demonstrate effectiveness of our contributions, we incrementally ablate
them over the baseline weakly-supervised SCG on V-COCO, HICO-DET and
Conceptual Captions. The results are shown in Tables 3, 4 and 5. While all of
our contributions clearly improve the performance over the baseline, results
also show that caption-dependent parts of our method (§Sec. 3.3 & §Sec. 3.5)
are not affected heavily from the caption source. Independent of whether
captions are collected in a controlled setting (V-COCO), scraped from the web
(Conceptual Captions) or generated by a captioning model (HICO-DET), our model
can utilize them to boost model performance.

Method | Sup. |  Agent AP ($\Delta$) |  Role AP ($\Delta$)
---|---|---|---
Baseline [41] (§3.1) | Weak | 32.41 | 20.05
+Pruning (§3.3) | Weak |  33.88 (+1.47) |  21.80 (+1.75)
+Suppressing (§3.4) | Weak |  37.04 (+4.63) |  28.28 (+8.23)
+Preposition (§3.5) | Weak |  40.53 (+8.12) |  29.59 (+9.54)
Table 3: Incremental ablations on V-COCO. $\Delta$ denotes performance difference over Baseline. All three of our contributions help improving HOI detection performance. Method | Sup. |  mAP ($\Delta$)
---|---|---
Baseline [41] (§3.1) | Weak | 7.05
+Pruning (§3.3) | Weak |  7.55 (+0.50)
+Suppressing (§3.4) | Weak |  7.81 (+0.76)
+Preposition (§3.5) | Weak |  8.38 (+1.33)
Table 4: Incremental ablations on HICO-DET. $\Delta$ denotes performance difference over Baseline. All three of our contributions help improving HOI detection performance. Method | Sup. |  Agent AP ($\Delta$) |  Role AP ($\Delta$)
---|---|---|---
Baseline [41] (§3.1) | Weak | 17.71 | 14.33
+Pruning (§3.3) | Weak |  19.44 (+1.73) |  15.95 (+1.62)
+Suppressing (§3.4) | Weak |  20.00 (+2.29) |  18.23 (+3.90)
+Preposition (§3.5) | Weak |  20.75 (+3.04) |  17.71 (+3.38)
Table 5: Incremental ablations on Conceptual Captions. $\Delta$ denotes
performance difference over Baseline. While all three contributions help
improve performance over Baseline, the preposition prediction task slightly
decreases Role AP when added on top of implausible interaction suppression
(but still boosts Agent AP).

##  5 Conclusion

In this work, we tackle HOI detection problem with the weakest supervision
setting in the literature, using image-level interaction labels only (e.g.
“ride”). We exploit the implicit grounding capability of a vision-language
model, in order to prune non-interacting human and object proposals. We
restrict our model’s output space to natural interactions only, querying a
large language model if a given $<$interactions, object$>$ is plausible. We
lastly formulate a weakly-supervised preposition prediction task to improve
spatial reasoning capability of our model explicitly. For the first time in
the literature, we learn an HOI detector on image-caption pairs, extracting
image-level interaction labels out of captions.

Ethical concerns. VLMs and LLMs can contain implicit biases inherited from
their training data. Even though their usage within this work’s context did
not pose any explicit harm during our experimentation, we would like to warn
users that their usage in a different context may expose people to potentially
unethical content.

## References

  * [1] Shuai Bai, Zhedong Zheng, Xiaohan Wang, Junyang Lin, Zhu Zhang, Chang Zhou, Hongxia Yang, and Yi Yang.  Connecting language and vision for natural language-based vehicle retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4034–4043, 2021.
  * [2] Hakan Bilen and Andrea Vedaldi.  Weakly supervised deep detection networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2846–2854, 2016.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 213–229. Springer, 2020.
  * [4] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558–3568, 2021.
  * [5] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng.  Learning to detect human-object interactions.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 381–389. IEEE, 2018.
  * [6] Hila Chefer, Shir Gur, and Lior Wolf.  Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 397–406, 2021.
  * [7] Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan, Sheng Zhong, and Xu Zou.  Category-aware transformer network for better human-object interaction detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19538–19547, 2022.
  * [8] Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, and Aleksandr Petiushko.  Mdmmt: Multidomain multimodal transformer for video retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3354–3363, 2021.
  * [9] Chen Gao, Yuliang Zou, and Jia-Bin Huang.  ican: Instance-centric attention network for human-object interaction detection.  In British Machine Vision Conference, 2018.
  * [10] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  In International Conference on Learning Representations, 2022.
  * [11] Saurabh Gupta and Jitendra Malik.  Visual semantic role labeling.  arXiv preprint arXiv:1505.04474, 2015.
  * [12] Tanmay Gupta, Alexander Schwing, and Derek Hoiem.  No-frills human-object interaction detection: Factorization, layout encodings, and training techniques.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9677–9685, 2019.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [14] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd.  spaCy: Industrial-strength Natural Language Processing in Python.  2020\.
  * [15] Yash Kant, Dhruv Batra, Peter Anderson, Alexander Schwing, Devi Parikh, Jiasen Lu, and Harsh Agrawal.  Spatially aware multimodal transformers for textvqa.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pages 715–732. Springer, 2020.
  * [16] Mert Kilickaya and Arnold WM Smeulders.  Human-object interaction detection without alignment supervision.  In British Machine Vision Conference, 2021.
  * [17] Bumsoo Kim, Taeho Choi, Jaewoo Kang, and Hyunwoo J Kim.  Uniondet: Union-level detector towards real-time human-object interaction detection.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16, pages 498–514. Springer, 2020.
  * [18] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim.  Hotr: End-to-end human-object interaction detection with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 74–83, 2021.
  * [19] Bumsoo Kim, Jonghwan Mun, Kyoung-Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim.  Mstr: Multi-scale transformer for end-to-end human-object interaction detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19578–19587, 2022.
  * [20] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon.  Detecting human-object interactions with action co-occurrence priors.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 718–736. Springer, 2020.
  * [21] Suresh Kirthi Kumaraswamy, Miaojing Shi, and Ewa Kijak.  Detecting human-object interaction with mixed supervision.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1228–1237, 2021.
  * [22] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu.  Hoi analysis: Integrating and decomposing human-object interaction.  Advances in Neural Information Processing Systems, 33:5011–5022, 2020.
  * [23] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu.  Transferable interactiveness knowledge for human-object interaction detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3585–3594, 2019.
  * [24] Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng.  Ppdm: Parallel point detection and matching for real-time human-object interaction detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 482–490, 2020.
  * [25] Y. Liao, A. Zhang, M. Lu, Y. Wang, X. Li, and S. Liu.  Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [26] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2980–2988, 2017.
  * [27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [28] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  ArXiv, abs/1907.11692, 2019.
  * [29] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei.  Visual relationship detection with language priors.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 852–869. Springer, 2016.
  * [30] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.  Neural baby talk.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7219–7228, 2018.
  * [31] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2085–2094, 2021.
  * [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [33] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 779–788, 2016.
  * [34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in Neural Information Processing Systems, 28, 2015.
  * [35] Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning.  Generating semantically precise scene graphs from textual descriptions for improved image retrieval.  In Proceedings of the Fourth Workshop on Vision and Language. Association for Computational Linguistics, 2015.
  * [36] Hengcan Shi, Munawar Hayat, Yicheng Wu, and Jianfei Cai.  Proposalclip: Unsupervised open-category object proposal generation via exploiting clip cues.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9611–9620, 2022.
  * [37] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317–8326, 2019.
  * [38] Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath.  Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13617–13626, 2020.
  * [39] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In International Conference on Machine Learning, pages 23318–23340. PMLR, 2022.
  * [40] Keren Ye and Adriana Kovashka.  Linguistic structures as weak supervision for visual scene graph generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8289–8299, June 2021.
  * [41] Frederic Z Zhang, Dylan Campbell, and Stephen Gould.  Spatially conditioned graphs for detecting human-object interactions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13319–13327, 2021.
  * [42] C Lawrence Zitnick and Piotr Dollár.  Edge boxes: Locating object proposals from edges.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 391–405. Springer, 2014.


# Padovan numbers which are concatenations of three repdigits

Kisan Bhoi and Prasanta Kumar Ray

###### Abstract

Padovan sequence is a ternary recurrent sequence defined by the recurrence
relation $P_{n+3}=P_{n+1}+P_{n}$ with initial terms $P_{0}=P_{1}=P_{2}=1.$ In
this study it is shown that
$114,151,200,265,351,465,616,816,3329,4410,7739,922111$ are the only Padovan
numbers which are concatenations of three repdigits.

Keywords:Padovan numbers, concatenations, repdigits, linear forms in
logarithms, reduction method.
2020 Mathematics Subject Classification: 11B39; 11J86; 11D61.

##  1 Introduction

A natural number $N$ is called a repdigit if it has only one distict digit in
its decimal expansion. That is, $N$ is of the form
$N=a\left(\frac{10^{m}-1}{9}\right)=\overline{\underbrace{a\ldots
a}_{m~{}\text{times}}}$ for $0\leq a\leq 9$ and $m\geq 1.$ A natural number of
the form

$N=\overline{\underbrace{a\ldots a}_{l~{}\text{times}}\underbrace{b\ldots
b}_{m~{}\text{times}}\underbrace{c\ldots c}_{k~{}\text{times}}}$

can be viewed as concatenation of three repdigits where $a,l,m,k\geq 1$ and
$0\leq a,b,c\leq 9.$ Let $\\{P_{n}\\}_{n\geq 0}$ be the Padovan sequence given
by

$P_{n+3}=P_{n+1}+P_{n}~{}\text{for all}~{}n\geq 1,$

with $P_{0}=P_{1}=P_{2}=1.$ The first few terms of this sequence are

$1,1,1,2,2,3,4,5,7,9,12,16,21,28,37,49,65,86,114,151,200,265,351,\ldots.$

In recent times, concatenation of repdigits in different linear recurrence
sequences has been studied by many researchers. For example, Alahmadi et al.
[1] studied the problem of finding all Fibonacci numbers which are
concatenations of two repdigits. They examined the same problem for
$k$-generalised Fibonacci sequence in [2]. Ddamulira searched all Padovan and
Tribonacci numbers which are concatenations of two repdigits in [5], [6],
respectively. Batte et al. [3] proved that the only Perrin numbers which are
concatenations of two repdigits are
$\\{10,12,17,22,29,39,51,68,90,119,277,644\\}$. In [14], Rayguru and Panda
showed that $35$ is the only balancing number which is concatenation of two
repdigits. Recently, Siar and Keskin [13] found that
$12,13,29,33,34,70,84,88,89,228$, and $233$ are the only $k$-generalized Pell
numbers, which are concatenations of two repdigits with at least two digits.
Erduvan and Keskin [9], in their work obtained Lucas numbers which are
concatenations of two repdigits. In [10], the same authors looked into Lucas
numbers which are concatenations of three repdigits. They proved that the only
Lucas numbers which are concatenations of three repdigits are
$123,199,322,521,843,2207,5778$. For the proof, they used lower bounds for
linear forms in logarithms and Baker-Davenport reduction method due to Dujella
and Pethő [8]. In this paper, we find all Padovan numbers that are
concatenations of three repdigits. More precisely, we completely solve the
Diophantine equation

$\displaystyle P_{n}$ $\displaystyle=$
$\displaystyle\overline{\underbrace{a\ldots
a}_{l~{}\text{times}}\underbrace{b\ldots
b}_{m~{}\text{times}}\underbrace{c\ldots c}_{k~{}\text{times}}}$
$\displaystyle=$ $\displaystyle\overline{\underbrace{a\ldots
a}_{l~{}\text{times}}}\cdot 10^{m+k}+\overline{\underbrace{b\ldots
b}_{m~{}\text{times}}}\cdot 10^{k}+\overline{\underbrace{c\ldots
c}_{k~{}\text{times}}}$ $\displaystyle=$ $\displaystyle
a\left(\frac{10^{l}-1}{9}\right)10^{m+k}+b\left(\frac{10^{m}-1}{9}\right)10^{k}+c\left(\frac{10^{k}-1}{9}\right),$

that is,

$P_{n}=\frac{1}{9}\left(a10^{l+m+k}-(a-b)10^{m+k}-(b-c)10^{k}-c\right).$ (1.1)

Our main result is the following.

######  Theorem 1.1.

The only Padovan numbers that are concatenations of three repdigits are
$114,151,200,265,351,465,616,816,3329,4410,7739,922111.$

For the proof of Theorem 1.1, we run a program in Mathemtica and search all
solutions of (1.1) with $n\leq 560$. Then, we take $n>560$ and write (1.1) in
three different ways to get three linear forms in logarithms. We apply lower
bounds for linear forms in logarithms to obtain an upper bound on $n.$ To
reduce the bound, we use Baker-Davenport reduction method de Weger [16]. As a
result, we get $n\leq 546,$ a contradiction. In this way, we complete the
proof of our main result.

We need some preliminary results which are discussed in the following section.

##  2 Preliminary Results

###  2.1 Some properties of Padovan sequence

The characteristic equation of $\\{P_{n}\\}_{n\geq 0}$ is $f(x)=x^{3}-x-1=0$
which has one real root $\alpha$ and two complex conjugate roots $\beta$ and
$\gamma$ given by

$\alpha=\frac{r_{1}+r_{2}}{6},~{}~{}\beta=\bar{\gamma}=\frac{-r_{1}-r_{2}+i\sqrt{3}(r_{1}-r_{2})}{12},$

where $r_{1}=\sqrt[3]{108+12\sqrt{69}}$ and $r_{2}=\sqrt[3]{108-12\sqrt{69}}$.
Binet’s formula for Padovan sequence is given by

$P_{n}=C_{\alpha}\alpha^{n}+C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n},$ (2.1)

where

$\displaystyle
C_{\alpha}=\frac{(1-\beta)(1-\gamma)}{(\alpha-\beta)(\alpha-\gamma)}=\frac{\alpha+1}{-\alpha^{2}+3\alpha+1},$
$\displaystyle
C_{\beta}=\frac{(1-\alpha)(1-\gamma)}{(\beta-\alpha)(\beta-\gamma)}=\frac{\beta+1}{-\beta^{2}+3\beta+1},$
$\displaystyle
C_{\gamma}=\frac{(1-\alpha)(1-\beta)}{(\gamma-\alpha)(\gamma-\beta)}=\frac{\gamma+1}{-\gamma^{2}+3\gamma+1}.$

The minimal polynomial of $C_{\alpha}$ over $\mathbb{Z}$ is
$23x^{3}-23x^{2}+6x-1$ with all its zeros of unit modulus. Numerically, we
have the following estimates:

$\displaystyle 1.32<\alpha<1.33,$ $\displaystyle
0.86<|\beta\rvert=|\gamma|=\alpha^{-1/2}<0.87,$ $\displaystyle
0.72<C_{\alpha}<0.73,$ $\displaystyle 0.24<|C_{\beta}|=|C_{\gamma}|<0.25.$

Let
$e(n):=P_{n}-C_{\alpha}\alpha^{n}=C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n}.$
Then $|e(n)|<\frac{1}{\alpha^{n/2}}$ for all $n\geq 1.$ Using induction, one
can prove that

$\alpha^{n-3}\leq P_{n}\leq\alpha^{n-1}~{}\text{holds for all}~{}n\geq 1.$
(2.2)

###  2.2 Linear forms in Logarithms

Baker’s theory plays an important role in reducing the bounds concerning
linear forms in logarithms of algebraic numbers. Let $\eta$ be an algebraic
number with minimal primitive polynomial

$\displaystyle
f\left(X\right)=a_{0}x^{d}+a_{1}x^{d-1}+\ldots+a_{d}=a_{0}\prod_{i=1}^{d}(X-\eta^{(i)})\in{\mathbb{Z}\left[X\right]},$

where the leading coefficient $a_{0}>0$, and $\eta^{(i)}$’s are conjugates of
$\eta$. Then, the logarithmic height of $\eta$ is given by

$\displaystyle h(\eta)=\frac{1}{d}\left(\log
a_{0}+\sum_{j=1}^{d}\text{max}\\{0,\log|\eta^{(j)}|\\}\right).$

The following are some properties of the logarithmic height function (see [15,
Property 3.3]), which will be used to calculate the heights of the algebraic
integers

$\displaystyle h(\eta+\gamma)\leq h(\eta)+h(\gamma)+\log 2,$ $\displaystyle
h(\eta\gamma^{\pm 1})\leq h(\eta)+h(\gamma),$ $\displaystyle
h(\eta^{k})=|k|h(\eta),k\in\mathbb{Z}.$

With these notations, Matveev (see [12] or [4, Theorem 9.4]) proved the
following result.

######  Theorem 2.1.

Let $\eta_{1},\eta_{2},\ldots,\eta_{l}$ be positive real algebraic integers in
a real algebraic number field $\mathbb{L}$ of degree $d_{\mathbb{L}}$ and
$b_{1},b_{2},\ldots,b_{l}$ be non zero integers. If
$\Gamma=\prod_{i=1}^{l}\eta_{i}^{b_{i}}-1$ is not zero, then

$\displaystyle\log|\Gamma|>-1.4\cdot 30^{l+3}l^{4.5}d_{\mathbb{L}}^{2}(1+\log
d_{\mathbb{L}})(1+\log D)A_{1}A_{2}\ldots A_{l},$

where $D=\text{max}\\{|b_{1}|,|b_{2}|,\ldots,|b_{l}|\\}$ and
$A_{1},A_{2},\ldots,A_{l}$ are positive real numbers such that

$\displaystyle
A_{j}\geq\text{max}\\{d_{\mathbb{L}}h\left(\eta_{j}\right),|\log\eta_{j}|,0.16\\}~{}~{}\text{for}~{}j=1,\ldots,l.$

###  2.3 Reduction method

The following Baker-Davenport reduction method due to de Weger [16] is used to
reduce the bounds of the variables which are too large. Let
$\vartheta_{1},\vartheta_{2},\beta\in\mathbb{R}$ be given and
$x_{1},x_{2}\in\mathbb{Z}$ be unknowns. Suppose

$\displaystyle\Lambda=\beta+x_{1}\vartheta_{1}+x_{2}\vartheta_{2}.$ (2.3)

Set $X=\max\\{|x_{1}|,|x_{2}|\\}$ and $X_{0},Y$ be positive. Assume that

$\displaystyle X\leq X_{0},$ (2.4)

and

$\displaystyle|\Lambda|<c\cdot\exp(-\delta\cdot Y),$ (2.5)

where $c,\delta$ be positive constants. When $\beta\neq 0$ in (2.3), put
$\vartheta=-\vartheta_{1}/\vartheta_{2}$ and $\psi=\beta/\vartheta_{2}$. Then
we have

$\displaystyle\frac{\Lambda}{\vartheta_{2}}=\psi-x_{1}\vartheta+x_{2}.$

Let $p/q$ be a convergent of $\vartheta$ with $q>X_{0}.$ For a real number
$x,$ we let $\|x\|=\min\\{|x-n|,n\in\mathbb{Z}\\}$ be the distance from $x$ to
the nearest integer. We have the following result.

######  Lemma 2.2.

([16, Lemma 3.3]) Suppose that

$\displaystyle\|q\psi\|>\frac{2X_{0}}{q}.$

Then, the solutions of (2.5) and (2.4) satisfy

$\displaystyle
Y<\frac{1}{\delta}\log\left(\frac{q^{2}c}{|\vartheta_{2}|X_{0}}\right).$

We state and prove the following lemma which gives a relation between $n$ amd
$l+m+k$ of (1.1).

######  Lemma 2.3.

All solutions to the Diophantine equation (1.1) satisfy

$(l+m+k)\log 10-3<n\log\alpha<(l+m+k)\log 10+1.$

###### Proof.

From (1.1) and (2.2) we get

$\alpha^{n-3}\leq P_{n}<10^{l+m+k}.$

Taking logarithm on both sides, we have

$(n-3)\log\alpha<(l+m+k)\log 10,$

which leads to

$n\log\alpha<(l+m+k)\log 10+3\log\alpha<(l+m+k)\log 10+1.$

On the other hand, for the lower bound, (1.1) implies that

$10^{l+m+k-1}<P_{n}\leq\alpha^{n-1}.$

Taking logarithm on both sides, we get

$(l+m+k-1)\log 10<(n-1)\log\alpha,$

which leads to

$(l+m+k-1)\log 10-3<(l+m+k-1)\log 10+\log\alpha<n\log\alpha.$

∎

The following result will also be used in our proof.

######  Lemma 2.4.

If $\lvert e^{z}-1\rvert<y<\frac{1}{2}$ for real values of $z$ and $y$, then
$\lvert z\rvert<2y$.

###### Proof.

If $z\geq 0,$ we have $0\leq z\leq e^{z}-1=\lvert e^{z}-1\rvert<y.$ If $z<0,$
then $\lvert e^{z}-1\rvert<\frac{1}{2}.$ From this, we get $e^{\lvert
z\rvert}<2,$ and therefore $0<\lvert z\rvert<e^{\lvert z\rvert}-1=e^{\lvert
z\rvert}\lvert e^{z}-1\rvert<2y$. so, in both cases, we get $\lvert
z\rvert<2y.$ ∎

The following lemma will be used in our proof. It is seen in [11, Lemma 7].

######  Lemma 2.5.

Let $r\geq 1$ and $H>0$ be such that $H>(4r^{2})^{r}$ and $H>L/(\log L)^{r}$.
Then

$\displaystyle L<2^{r}H(\log H)^{r}.$

##  3 Proof of the main result

Using Mathematica, we find all solutions to the Diophantine equation (1.1) for
$a\geq 1$, $0\leq a,b,c\leq 9$ and $1\leq n\leq 560$ that are listed in
Theorem 1.1. Now, assume that $n>560.$ We ignore the cases $a=b\neq c$ and
$a\neq b=c$ since these cases have been treated in [5]. We also ignore the
case $a=b=c$ as it has been studied in [7]. Using (1.1) and (2.1), we have

$C_{\alpha}\alpha^{n}+C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n}=\frac{1}{9}\left(a10^{l+m+k}-(a-b)10^{m+k}-(b-c)10^{k}-c\right).$
(3.1)

We can write (3.1)as

$9C_{\alpha}\alpha^{n}-a10^{l+m+k}=-9(C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n})-(a-b)10^{m+k}-(b-c)10^{k}-c.$
(3.2)

Taking absolute values on both sides of (3.2), we have that

$\displaystyle\left|9C_{\alpha}\alpha^{n}-a10^{l+m+k}\right|$
$\displaystyle\leq
9\left|(C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n})\right|+\lvert(a-b)\rvert
10^{m+k}+\lvert(b-c)\rvert 10^{k}+\lvert c\rvert$ $\displaystyle\leq
9\alpha^{-n/2}+9\cdot 10^{m+k}+9\cdot 10^{k}+9$ $\displaystyle<11\cdot
10^{m+k}$

Dividing both sides of the above inequality by $a10^{l+m+k}$ gives

$\left|\left(\frac{9C_{\alpha}}{a}\right)\alpha^{n}10^{-(l+m+k)}-1\right|<\frac{11\cdot
10^{m+k}}{a10^{l+m+k}}<\frac{11}{10^{l}}.$ (3.3)

Let

$\Gamma_{1}=\left(\frac{9C_{\alpha}}{a}\right)\alpha^{n}10^{-(l+m+k)}-1.$

Note that $\Gamma_{1}\neq 0.$ If $\Gamma_{1}=0,$ then

$C_{\alpha}\alpha^{n}=\frac{a}{9}10^{l+m+k}.$ (3.4)

Let $\sigma$ be the automorphism of the Galois group of the splitting field of
$f(x)$ over $\mathbb{Q}$ defined by $\sigma(\alpha)=\beta,$ where
$f(x)=x^{3}-x-1$ is the minimal polynomial of $\alpha$. Applying $\sigma$ on
both sides of (3.4), and taking absolute values, we get

$\displaystyle\lvert
C_{\beta}\beta^{n}\rvert=\left|\left(\frac{a}{9}10^{l+m+k}\right)\right|,$

which is not possible since $\lvert C_{\beta}\beta^{n}\rvert<1$, whereas
$\left|\left(\frac{a}{9}10^{l+m+k}\right)\right|>1.$ Therefore,
$\Gamma_{1}\neq 0.$ Now, we are ready to apply Theorem 2.1 with the following
data:

$\displaystyle\eta_{1}=\frac{9C_{\alpha}}{a},~{}\eta_{2}=\alpha,~{}\eta_{3}=10,~{}b_{1}=1,~{}b_{2}=n,~{}b_{3}=-(l+m+k),~{}l=3,$

with $d_{\mathbb{L}}=[\mathbb{Q}(\alpha):\mathbb{Q}]=3.$ Since $l+m+k<n,$ take
$D=n$. The heights of $\eta_{1},\eta_{2},\eta_{3}$ are calculated as follows:

$h(\eta_{1})=h(9C_{\alpha}/a)\leq h(9)+h(C_{\alpha})+h(a)\leq 2\log
9+\frac{\log 23}{3}<5.44,$
$h(\eta_{2})=h(\alpha)=\frac{\log\alpha}{3},~{}h(\eta_{3})=h(10)=\log 10.$

Thus, we take

$A_{1}=16.32,~{}~{}~{}A_{2}=\log\alpha,~{}~{}~{}~{}A_{3}=3\log 10.$

Applying Theorem 2.1 we find

$\displaystyle\log|\Gamma_{1}|$ $\displaystyle>-1.4\cdot
30^{6}3^{4.5}3^{2}(1+\log 3)(1+\log n)(16.32)(\log\alpha)(3\log 10)$
$\displaystyle>-8.58\cdot 10^{13}\log(1+\log n).$

Comparing the above inequality with (3.3) gives

$\displaystyle l\log 10-\log 11<8.58\cdot 10^{13}(1+\log n),$

which leads to

$l\log 10<8.59\cdot 10^{13}(1+\log n).$ (3.5)

Rewriting (3.1) in another way, we have

$9C_{\alpha}\alpha^{n}-(a10^{l}-(a-b))10^{m+k}=-9(C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n})-(b-c)10^{k}-c.$
(3.6)

Taking absolute values on both sides of (3.6), we obtain

$\displaystyle\left|9C_{\alpha}\alpha^{n}-(a10^{l}-(a-b))10^{m+k}\right|$
$\displaystyle\leq
9|(C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n})|+\lvert(b-c)\rvert 10^{k}+\lvert
c\rvert$ $\displaystyle\leq 9\alpha^{-n/2}+9\cdot 10^{k}+9$
$\displaystyle<11\cdot 10^{k}$

Dividing both sides by $(a10^{l}-(a-b))10^{m+k}$, we obtain

$\left|1-\left(\frac{9C_{\alpha}}{a10^{l}-(a-b)}\right)\alpha^{n}10^{-(m+k)}\right|<\frac{11\cdot
10^{k}}{(a10^{l}-(a-b))10^{m+k}}<\frac{11}{10^{m}}.$ (3.7)

Put

$\Gamma_{2}=1-\left(\frac{9C_{\alpha}}{a10^{l}-(a-b)}\right)\alpha^{n}10^{-(m+k)}.$

By using similar reason as above we can show that $\Gamma_{2}\neq 0$. Now, we
apply Theorem 2.1 with the following parameters:

$\displaystyle\eta_{1}=\frac{9C_{\alpha}}{a10^{l}-(a-b)},~{}\eta_{2}=\alpha,~{}\eta_{3}=10,~{}b_{1}=1,~{}b_{2}=n,~{}b_{3}=-(m+k),~{}l=3,$

with $d_{\mathbb{L}}=[\mathbb{Q}(\alpha):\mathbb{Q}]=3.$ Since $m+k<n,$ take
$D=n$. Using the properties of logarithmic height, we estimate $h(\eta_{1})$
as follows:

$\displaystyle h(\eta_{1})$
$\displaystyle=h\left(\frac{9C_{\alpha}}{a10^{l}-(a-b)}\right)\leq
h(9C_{\alpha})+h\left(a10^{l}-(a-b)\right)$ $\displaystyle\leq
h(9)+h(C_{\alpha})+h(a10^{l})+h(a-b)+\log 2$ $\displaystyle\leq
h(9)+h(C_{\alpha})+h(a)+lh(10)+h(a)+h(b)+2\log 2$ $\displaystyle\leq 4\log
9+l\log 10+\frac{\log 23}{3}+2\log 2$ $\displaystyle<11.23+l\log 10.$

Hence from (3.5), we get

$h(\eta_{1})<11.23+8.59\cdot 10^{13}(1+\log n)<8.6\cdot 10^{13}(1+\log n).$

Furthermore, $h(\eta_{2})=h(\alpha)=\frac{\log\alpha}{3}$ and
$h(\eta_{3})=h(10)=\log 10.$ Thus, we take

$A_{1}=25.8\cdot 10^{13}(1+\log
n),~{}~{}~{}A_{2}=\log\alpha,~{}~{}~{}~{}A_{3}=3\log 10.$

By virtue of Theorem 2.1 we have

$\displaystyle\log|\Gamma_{2}|$ $\displaystyle>-1.4\cdot
30^{6}3^{4.5}3^{2}(1+\log 3)(1+\log n)(25.8\cdot 10^{13}(1+\log
n))(\log\alpha)(3\log 10)$ $\displaystyle>-1.36\cdot 10^{27}(1+\log n)^{2}.$

Comparing the above inequality with (3.7) gives

$\displaystyle m\log 10-\log 11<1.36\cdot 10^{27}(1+\log n)^{2},$

which leads to

$m\log 10<1.37\cdot 10^{27}(1+\log n)^{2}.$ (3.8)

Again, rearranging (3.1), we have

$9C_{\alpha}\alpha^{n}-(a10^{l+m}-(a-b)10^{m}-(b-c))10^{k}=-9(C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n})-c.$
(3.9)

Taking absolute values on both sides of (3.9), we obtain

$\displaystyle\left\lvert
9C_{\alpha}\alpha^{n}-(a10^{l+m}-(a-b)10^{m}-(b-c))10^{k}\right\rvert$
$\displaystyle\leq 9|(C_{\beta}\beta^{n}+C_{\gamma}\gamma^{n})|+\lvert
c\rvert$ $\displaystyle\leq 9\alpha^{n/2}+9<10.$

Dividing both sides by $9C_{\alpha}\alpha^{n}$, we obtain

$\left\lvert
1-\left(\frac{a10^{l+m}-(a-b)10^{m}-(b-c)}{9C_{\alpha}}\right)\alpha^{-n}10^{k}\right\rvert<\frac{10}{9C_{\alpha}\alpha^{n}}<\frac{2.5}{\alpha^{n}}.$
(3.10)

Let

$\Gamma_{3}=1-\left(\frac{a10^{l+m}-(a-b)10^{m}-(b-c)}{9C_{\alpha}}\right)\alpha^{-n}10^{k}.$

By using similar arguments as before it is seen that $\Gamma_{3}\neq 0$. Now,
we apply Theorem 2.1 with the following parameters:

$\displaystyle\eta_{1}=\frac{a10^{l+m}-(a-b)10^{m}-(b-c)}{9C_{\alpha}},~{}\eta_{2}=\alpha,~{}\eta_{3}=10,~{}b_{1}=1,~{}b_{2}=-n,~{}b_{3}=k,~{}l=3,$

with $d_{\mathbb{L}}=[\mathbb{Q}(\alpha):\mathbb{Q}]=3.$ Since $k<n,$ take
$D=n$. Using the properties of logarithmic height, we estimate $h(\eta_{1})$
as follows:

$\displaystyle h(\eta_{1})$
$\displaystyle=h\left(\frac{a10^{l+m}-(a-b)10^{m}-(b-c)}{9C_{\alpha}}\right)$
$\displaystyle\leq h(a10^{l+m})+h((a-b)10^{m})+h(b-c)+h(9C_{\alpha})+2\log 2$
$\displaystyle\leq
h(a)+h(10^{l+m})+h(a-b)+h(10^{m})+h(b-c)+h(9)+h(C_{\alpha})+2\log 2$
$\displaystyle\leq
h(9)+(l+m)h(10)+h(a)+h(b)+mh(10)+h(b)+h(c)+h(9)+h(C_{\alpha})+4\log 2$
$\displaystyle\leq 6\log 9+(l+m)\log 10+m\log 10+\frac{\log 23}{3}+4\log 2$
$\displaystyle<17.1+l\log 10+2m\log 10.$

Hence from (3.5) and (3.8), we get

$h(\eta_{1})<17.1+8.59\cdot 10^{13}(1+\log n)+2.74\cdot 10^{27}(1+\log
n)^{2}<2.76\cdot 10^{27}(1+\log n)^{2}.$

Furthermore, $h(\eta_{2})=h(\alpha)=\frac{\log\alpha}{3}$ and
$h(\eta_{3})=h(10)=\log 10.$ Thus, we take

$A_{1}=8.28\cdot 10^{27}(1+\log
n)^{2},~{}~{}~{}A_{2}=\log\alpha,~{}~{}~{}~{}A_{3}=3\log 10.$

Using Theorem 2.1 we have

$\displaystyle\log|\Gamma_{3}|$ $\displaystyle>-1.4\cdot
30^{6}3^{4.5}3^{2}(1+\log 3)(1+\log n)\left(8.28\cdot 10^{27}(1+\log
n)^{2}\right)(\log\alpha)(3\log 10)$ $\displaystyle>-4.35\cdot 10^{48}(1+\log
n)^{3}.$

Comparing the above inequality with (3.10) gives

$\displaystyle n\log\alpha-\log 2.5<4.35\cdot 10^{48}(1+\log n)^{3},$

which further implies

$n<15.6\cdot 10^{48}(1+\log n)^{3}.$

With the notation of Lemma 2.5, we take $r=3$, $L=n$ and $H=15.6\cdot
10^{48}$. Applying Lemma 2.5, we have

$\displaystyle n$ $\displaystyle<2^{3}(15.6\cdot 10^{48})(\log(15.6\cdot
10^{48})^{3}$ $\displaystyle<2\cdot 10^{56}.$

Now, to reduce the bounds, let

$\Lambda_{1}=(l+m+k)\log
10-n\log\alpha-\log\left(\frac{9C_{\alpha}}{a}\right).$

The inequality (3.3) can be written as

$\lvert e^{-\Lambda_{1}}-1\rvert<\frac{11}{10^{l}}.$

Notice that $-\Lambda_{1}\neq 0$ as $e^{-\Lambda_{1}}-1=\Gamma_{1}\neq 0.$
Assuming $l\geq 2$, the right-hand side in the above inequality is at most
$\frac{1}{2}.$ Using Lemma 2.4 we obtain

$0<\lvert\Lambda_{1}\rvert<\frac{22}{10^{l}},$

which implies that

$\left\lvert(l+m+k)\log
10-n\log\alpha-\log\left(\frac{9C_{\alpha}}{a}\right)\right\rvert<22\exp(-l\log
10).$

We also have

$\displaystyle\frac{\Lambda_{1}}{\log
10}=(l+m+k)-n\left(\frac{\log\alpha}{\log
10}\right)-\frac{\log(9C_{\alpha}/a)}{\log 10}.$

Thus, we take

$\displaystyle c=22,~{}\delta=\log 10,~{}X_{0}=2\cdot
10^{56},~{}\psi=-\frac{\log(9C_{\alpha}/a)}{\log 10}$
$\displaystyle\vartheta=\frac{\log\alpha}{\log
10},~{}\vartheta_{1}=-\log\alpha,~{}\vartheta_{2}=\log
10,~{}\beta=-\log\left(\frac{9C_{\alpha}}{a}\right).$

We find that
$\frac{p_{121}}{q_{121}}=\frac{17548495448098062534665425097069587403994668418373610729747}{143694755301644024543505669827725455817494147218758974051600}$
is the $121$th convergent of $\vartheta$ such that $q_{121}>X_{0}$ and it
satisfies $\|q\psi\|>\frac{2X_{0}}{q}$. Applying Lemma 2.2, we get
$l<\frac{1}{\log
10}\log\left(\frac{143694755301644024543505669827725455817494147218758974051600^{2}\cdot
22}{\log 10\cdot 2\cdot 10^{56}}\right)\leq 62$. Now, let

$\Lambda_{2}=(m+k)\log
10-n\log\alpha-\log\left(\frac{9C_{\alpha}}{a10^{l}-(a-b)}\right).$

From (3.7), it is seen that

$\lvert e^{-\Lambda_{2}}-1\rvert<\frac{11}{10^{m}}.$

Notice that $\Lambda_{2}\neq 0$ as $e^{-\Lambda_{2}}-1=\Gamma_{2}\neq 0.$
Assuming $m\geq 2$, the right-hand side in the above inequality is at most
$\frac{1}{2}.$ Using Lemma 2.4 we obtain

$0<\lvert\Lambda_{2}\rvert<\frac{22}{10^{m}},$

which implies that

$\left\lvert(m+k)\log
10-n\log\alpha-\log\left(\frac{9C_{\alpha}}{a10^{l}-(a-b)}\right)\right\rvert<22\exp(-m\log
10).$

We also have

$\displaystyle\frac{\Lambda_{2}}{\log 10}=(m+k)-n\left(\frac{\log\alpha}{\log
10}\right)-\frac{\log(9C_{\alpha}/(a10^{l}-(a-b)))}{\log 10}.$

We choose

$\displaystyle c=22,~{}\delta=\log 10,~{}X_{0}=2\cdot
10^{56},~{}\psi=-\frac{\log(9C_{\alpha}/(a10^{l}-(a-b)))}{\log 10}$
$\displaystyle\vartheta=\frac{\log\alpha}{\log
10},~{}\vartheta_{1}=-\log\alpha,~{}\vartheta_{2}=\log
10,~{}\beta=-\log\left(\frac{9C_{\alpha}}{a10^{l}-(a-b)}\right).$

We find that
$\frac{p_{124}}{q_{124}}=\frac{627183116477915636965979019027840650348169004436382793310275}{5135649646898035023105055510310316619786462973990152740408498}$
is the $124$th convergent of $\vartheta$ such that $q_{124}>X_{0}$ and it
satisfies $\|q\psi\|>\frac{2X_{0}}{q}$. Using Lemma 2.2, we get $m\leq 66$.
Now, let

$\Lambda_{3}=k\log
10-n\log\alpha+\log\left(\frac{a10^{l+m}-(a-b)10^{m}-(b-c)}{9C_{\alpha}}\right).$

From (3.10), it is seen that

$\lvert e^{\Lambda_{3}}-1\rvert<\frac{2.5}{\alpha^{n}}.$

Notice that $\Lambda_{3}\neq 0$ as $e^{\Lambda_{3}}-1=\Gamma_{3}\neq 0.$ Since
$n>560$, the right-hand side in the above inequality is at most $\frac{1}{2}.$
Using Lemma 2.4 we obtain

$0<\lvert\Lambda_{3}\rvert<\frac{5}{\alpha^{n}},$

which implies that

$\left\lvert k\log
10-n\log\alpha+\log\left(\frac{a10^{l+m}-(a-b)10^{m}+(b-c)}{9C_{\alpha}}\right)\right\rvert<5\exp(-n\log\alpha).$

We also have

$\displaystyle\frac{\Lambda_{3}}{\log 10}=k-n\left(\frac{\log\alpha}{\log
10}\right)+\frac{\log((a10^{l+m}-(a-b)10^{m}+(b-c))/9C_{\alpha})}{\log 10}.$

Thus, we take

$\displaystyle c=5,~{}\delta=\log\alpha,~{}X_{0}=2\cdot
10^{56},~{}\psi=\frac{\log((a10^{l+m}-(a-b)10^{m}+(b-c))/9C_{\alpha})}{\log
10}$ $\displaystyle\vartheta=\frac{\log\alpha}{\log
10},~{}\vartheta_{1}=-\log\alpha,~{}\vartheta_{2}=\log
10,~{}\beta=\log\left(\frac{a10^{l+m}-(a-b)10^{m}+(b-c)}{9C_{\alpha}}\right).$

We find that
$\frac{p_{125}}{q_{125}}=\frac{2616187734565998164447356816798847641018036629887990573914437}{21422489321292086600361648904597606825079844749495429580710675}$
is the $125$th convergent of $\vartheta$ such that $q_{125}>X_{0}$ and it
satisfies $\|q\psi\|>\frac{2X_{0}}{q}$. Applying Lemma 2.2, we get $n\leq
546$, which is a contradiction to our assumption that $n>560$. Hence, the
theorem is proved.

## References

  * [1] A. Alahmadi, A. Altassan, F. Luca and H. Shoaib, Fibonacci numbers which are concatenations of two repdigits, Quaest. Math., 44 (2021), 281–290.
  * [2] A. Alahmadi, A. Altassan, F. Luca and H. Shoaib, $k$-generalized Fibonacci numbers which are concatenation of two repdigits, Glansik Mathemati$\check{c}$ki, 56 (2021), 29–46.
  * [3] H. Batte, T. P. Chalebgwa and M. Ddamulira, Perrin numbers that are concatenations of two repdigits, Arab. J. Math. 11 (2022), 469–478.
  * [4] Y. Bugeaud, M. Mignotte and S.Siksek, Classical and modular approaches to exponential Diophantine equations I. Fibonacci and Lucas perfect powers, Ann. of Math. (2) 163 (2006), 969–1018.
  * [5] M. Ddamulira, Padovan numbers that are concatenations of two repdigits, Math. Slovaca, 71 (2021), 275–284.
  * [6] M. Ddamulira, Tribonacci numbers that are concatenations of two repdigits, Rev. Acad. Colombiana Cienc. Exact. F$\acute{i}$s Natur., 114 (2020), 1–10.
  * [7] M. Ddamulira, Repdigits as sums of three Padovan numbers, Bol. Soc. Mat. Mex., 26 (2020), 247–261.
  * [8] A. Dujella and A. Pethő, A generalization of a theorem of Baker and Davenport, _Quart. J. Math. Oxford Ser._ , 49 (1998), 291-306.
  * [9] F. Erduvan and R. Keskin, Lucas numbers which are concatenations of two repdigits, Bol. Soc. Mat. Mex., 27 (2021), 1–11.
  * [10] F. Erduvan and R. Keskin, Lucas numbers which are concatenations of three repdigits, Results Math., 76 (2021), 1–13.
  * [11] S. G$\acute{\text{u}}$zman S$\acute{\text{a}}$nchez and F. Luca, Linear combinations of factorials and $s$-units in a binary recurrence sequence, Ann. Math. du Qu$\acute{\text{e}}$., 38 (2014), 169–188.
  * [12] E. M. Matveev, An explicit lower bound for a homogeneous rational linear form in the logarithms of algebraic numbers II, Izv. Ross. Akad. Nauk Ser. Mat., 64 (2000), 125–180. Translation in Izv. Math., 64 (2000), 1217–1269.
  * [13] Z. Şiar and R. Keskin, $k$-generalized Pell numbers which are concatenation of two repdigits, Mediterr. J. Math. 19 (2022), 1–17.
  * [14] S. G. Rayguru and G. K. Panda, Balancing numbers which are concatenation of two repdigits, Bol. Soc. Mat. Mex. 26 (2020), 911–919.
  * [15] M. Waldshmidt, Diophantine Approximation on Linear Algebraic Groups: Transcendence Properties of the Exponential Function in Several Variables, Springer-verlag, Berlin Heidel-berg, Berlin, 2000.
  * [16] B.M.M. de Weger, Algorithms for Diophantine Equations, Stichting Mathematisch Centrum, Amsterdam 1989.

Department of Mathematics
---
Sambalpur University, Jyoti Vihar, Burla, India
<EMAIL_ADDRESS>

Department of Mathematics
---
Sambalpur University, Jyoti Vihar, Burla, India
<EMAIL_ADDRESS>