# Crowd Counting in Harsh Weather using Image Denoising with Pix2Pix GANs
††thanks: This publication was made possible by the PDRA award
PDRA7-0606-21012 from the Qatar National Research Fund (a member of The Qatar
Foundation) and Qatar University Internal Grant No. IRCC-2023-237. The
statements made herein are solely the responsibility of the authors.

Muhammad Asif Khan, , Ridha Hamila., and Hamid Menouar  Muhammad Asif Khan1,
Hamid Menouar1, and Ridha Hamila2  Qatar Mobility Innovations Center, Qatar
University. 1 Department of Electrical Engineering, Qatar University2
{muhammada<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Index Terms:

crowd counting, CNN, density estimation, GAN, Pix2Pix

Visual crowd counting estimates the density of the crowd using deep learning
models such as convolution neural networks (CNNs). The performance of the
model heavily relies on the quality of the training data that constitutes
crowd images. In harsh weather such as fog, dust, and low light conditions,
the inference performance may severely degrade on the noisy and blur images.
In this paper, we propose the use of Pix2Pix generative adversarial network
(GAN) to first denoise the crowd images prior to passing them to the counting
model. A Pix2Pix network is trained using synthetic noisy images generated
from original crowd images and then the pretrained generator is then used in
the inference engine to estimate the crowd density in unseen, noisy crowd
images. The performance is tested on JHU-Crowd dataset to validate the
significance of the proposed method particularly when high reliability and
accuracy are required.

##  I Introduction

Crowd density estimation is a task in computer vision that aims to estimate
crowd density and counting using crowd images and videos. Crowd counting has
applications in several domains such as public safety, event planning,
transportation, security & surveillance, resource allocation, urban planning,
healthcare, and environmental monitoring. Although several methods have been
applied traditionally [1, 2, 3, 4], convolution neural network (CNN) is the
most widely used and effective approach. A CNN-based crowd-counting model is
trained using a dataset of annotated crowd images. The images are point
annotated by placing a dot on all head positions in the image. This produces a
sparse localization map i.e., a matrix of zeros and ones where one represents
the head position. To train the model, the localization map is transformed
into a density map by applying a Gaussian kernel to the head position. The
crowd model then regresses the target density maps. Over the years, several
state-of-the-art crowd-counting CNN architectures are proposed [5]. Some
examples include CrowdCNN [6], MCNN [7], CMTL [8], CSRNet [9], TEDnet [10],
SANet [11], SASNet [12], MobileCount [13], LCDnet [14], and DroneNet [15].
These models are developed using numerous architectural improvements and
training methods, resulting in incremental gains in counting accuracy over
benchmark crowd counting datasets.

Nevertheless, the prediction performance of these models degrades when the
crowd images are blurry and noisy. Noisy crowd images are common in many
practical scenarios such as during rain, fog, dust storms, object motion, etc.
When high accuracy is desired, it is inevitable that the crowd model is highly
robust against such image distortions. To provide highly accurate counting
performance, this work proposes to enhance the crowd image prior to passing it
through the counting model. Although traditional image enhancement methods can
be used for image denoising, more powerful methods such as generative
adversarial networks (GANs) [16] offer superior capabilities for image
enhancement over several image attributes (such as brightness, pixel
distortion, illumination, contrast, etc.) and produces a clearer and enhanced
version of the noisy crowd image. Pix2Pix [17] is a general-purpose image
translation GAN model used in many tasks such as map-to-aerial photos, black
and white-to-color photos, sketch-to-color photos, day-to-night photos,
thermal-color, semantic-to-photos, etc. This paper proposes the use of Pix2Pix
GANs for crowd image denoising and enhancement prior to passing to the crowd
model for density estimation. Pix2Pix GANs have been applied previously in
various image translation tasks [18, 19, 20] with significant improvements in
results.

The contribution of this paper is as follows:

  * •

A two-stage crowd density estimation method is proposed for applications
requiring high accuracy and robustness against extreme weather and
environmental conditions using integrating a Pix2Pix GAN and a density
estimation model.

  * •

A general-purpose Pix2Pix GAN network is trained over crowd datasets of pairs
of (clear and noisy) images. We synthetically added several noise and
distortion effects to mimic real-world harsh weather scenarios.

  * •

The pretrained generator of the Pix2Pix GAN model is then used as a
preprocessor in several crowd density estimation models. The prediction
performance of the GAN-aided crowd-counting models is compared with the
standard counting crowd models to validate the significance of the proposed
scheme.

##  II Related Work

Crowd counting using convolution neural networks (CNNs) was first introduced
in [6] using a simple architecture. Following this, several lightweight
architectures were proposed in [7, 21, 8, 22]. These models achieved
incremental gains in accuracy over benchmark datasets. However, significant
gains were achieved using crowd architectures using pretrained frontend
networks. These include CSRNet [9], CANNet [23], GSP [24], Deepcount [25],
SASNet, and SGANet [26].

Generative adversarial network (GAN) was first introduced in [16] as a two-
stage network that is comprised of two neural networks, a generator network,
and a discriminator network. The generator is trained to produce images in the
target domain, and the discriminator judges these images and classifies them
as real or fake. The end goal of the GAN is to make the generator output as
realistic (similar to the target images) as possible. There have been several
types of GAN architectures such as conditional GANs (cGAN) [27], DCGANs [28],
CycleGANs [29], StyleGAN [30], and Pix2Pix GAN [17].

Pix2Pix GANs are general-purpose GANs used in various image translation tasks
such as sketch-to-photograph conversion [31], image dehazing [32], depth image
generation [33], and computerized tomography (CT) scan image generation [19].
They have been used in image denoising in several studies. Authors in [34] use
a Pix2Pix GAN to denoise ultrasound images. Speckle noise using Rayleigh
distribution was added to the training images to generate image pairs of
original and noisy images. Authors in [35] use a Pix2Pix GAN with a penalty
term to render greyscale images into color images. The performance is
evaluated over PSNR and SSIM metrics. A pretrained Pix2Pix GAN is used in [36]
to denoise degraded electronic documents. The architecture replaces the U-net
network in the generator with ResNet6, and patchGAN in the discriminator
network. Similarly, [37] uses a Pix2Pix GAN with a residual network-based
generator and a PatchGAN-based discriminator to enhance CT scan images. Other
works on using Pix2Pix GAN for image denoising include [38, 39, 40].

##  III Proposed Scheme

We used a Pix2Pix GAN which denoises blurry crowd images to acquire clear
images which are then passed to a crowd density estimation model. The Pix2Pix
GAN is trained using pair of noisy (input) and clear (target) crowd images

###  III-A The Pix2Pix GAN architecture

The Pix2Pix GAN consists of a generator and a discriminator network. The
generator consists of a UNet architecture. Both the generator and the
discriminator consist of Convolution-Batch Normalization-ReLU blocks.

Figure 1: Image Denoising using Pix2Pix GAN. The images with gray borders are
input images, images with black borders are target images, and the image with
the green border is the generator’s prediction. G is the generator and D is
the discriminator of the Pix2Pix GAN.

The generator network translates the noisy image into an enhanced image as the
input image. The discriminator attempts to classify the generator output as
real or fake. The generator network uses$L_{1}$ or $L_{2}$ loss function as
follows:

$\mathcal{L}_{\text{L1}}(G)=\frac{1}{N}\sum_{i=1}^{N}\left|G(x_{i})-y_{i}\right|$
(1)

where $G$ is the generator network, $N$ is number of samples, $x_{i}$ is the
input sample, and $y_{i}$ is the ground truth value.

The discriminator network uses binary cross entropy (BCE) loss or adversarial
loss to measure the similarity between the discriminator’s predictions for the
generator’s output and the target image. It is calculated as Eq. 2:

$\displaystyle\mathcal{L}_{\text{BCE}}(D,G)$
$\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\bigg{(}y_{i}\log D(x_{i})$ (2)
$\displaystyle\quad+(1-y_{i})\log(1-D(G(x_{i})))\bigg{)}$

where $D$ is the discriminator network, $N$ is the total number of samples,
$D_{(x_{i})}$ is discriminator’s prediction for input $x_{i}$, and
$D(G_{(x_{i})})$ is discriminator’s prediction for the generated output
$G_{(x_{i})}$.

##  IV Experiments and Results

###  IV-A Datasets and Baselines

To train the Pix2Pix GAN, we use the ShanghaiTech dataset [7]. The dataset
contains a total of 1198 images and 330,165 annotations. The dataset is
divided into two parts. Part A contains 482 images and part B contains 716
images. Fig. 2 presents the histogram of per-image counts in the JHU-Crowd++
whereas Fig. 3 shows some representative samples from the subset.

Figure 2: Histogram of head counts in the JHU-Crowd dataset. Figure 3: Noisy
image samples from JHU-Crowd dataset showing harsh weather conditions.

As pairs of images (input and target images) are required to train the GAN,
random image distortions are applied to the original images such that the
original image is the target image whereas the distorted image is the input
image. The pretrained model is then applied to JHU-Crowd++ [41] dataset to
estimate crowd counts. The JHU-Crowd++ dataset contains 4,372 images and 1.51
million annotations. The dataset further contains additional attributes such
as adverse weather and illumination conditions. We used only the images with
adverse weather conditions to test the proposed method in this paper.

The performance is validated using four crowd counting models including MCNN
[7], CMTL [8], CSRNet [9], TEDnet [10], and SANet [11]. MCNN is a lightweight
multicolumn network consisting of three columns with a fusion layer at the
end. CSRNet is a deep architecture using the pretrained VGG-16 frontend (first
10 layers). TEDnet is an encoder-decoder network. SASNet is a scale-
aggregation network using a composite loss function i.e., a combination of
Euclidean loss and local pattern consistency loss.

###  IV-B Evaluation Metrics

Structural similarity index measure (SSIM), and pixel signal-to-noise ratio
(PSNR) are two standard metrics to compare the performance of image
translation using GANs (including Pix2Pix) model. PSNR measures the difference
between the corresponding pixels in the target image (label) and the predicted
image using GAN. SSIM measures the difference between corresponding pixels in
terms of brightness, contrast, and structure. These are defined as follows:

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{x}\sigma_{y}C_{2})}{(\mu_{z}^{2}\mu_{y}^{2}+C_{1})(\mu_{z}^{2}\mu_{y}^{2}+C_{2})}$
(3)

where $\mu_{x},\mu_{y},\sigma_{x},\sigma_{y}$ represent the means and standard
deviations of the actual and predicted density maps, respectively.

$PSNR=10log_{10}\left(\frac{Max(I^{2})}{MSE}\right)$ (4)

where $Max(I^{2})$ is the maximal in the image data. If it is an 8-bit
unsigned integer data type, the $Max(I^{2})=255$.

To measure the counting performance, we use the standard metric i.e., the mean
absolute error (MAE) which is calculated as follows:

$MAE=\frac{1}{N}\sum_{1}^{N}{(e_{n}-\hat{g_{n}})}$ (5)

###  IV-C Training

The Pix2Pix GAN model is trained for 500 epochs using a batch size of 32 and
the Adam optimizer with learning rates of $2e-4$ for both generator and
discriminator networks. A dropout of 0.5 was used in some layers of the
generator network. The crowd models are trained for 200 epochs using a batch
size of 16 and the Adam optimizer with a base learning rate of $0.0001$.

###  IV-D Results and Discussion

This section presents the experimental results of Pix2Pix GAN for image
denoising and the crowd counting performance using the baseline models and our
proposed scheme. The Pix2Pix GAN model trained on ShanghaiTech dataset is
evaluated on the JHU-Crowd++ dataset.

Fig. 4 shows samples from the training data for the Pix2Pix GAN. The training
images are synthetically generated by applying various noise types (Gaussian
noise, Gaussian blurs, etc.) to original images (serve as labels for the GAN).

Figure 4: Samples from training data (ShanghaiTech Part A) dataset for Pix2Pix
GAN. The top row contains noisy images, the middle row contains target images,
and the bottom row contains the corresponding density maps (density maps are
not used in GAN training).

Fig. 5 shows a visual illustration of image denoising on a single example
image before and after passing through the GAN model. The difference in the
noise level is very clear when zoomed in.

Figure 5: Illustration of image denoising using Pix2Pix GAN. The image on the
left shows the input image (noisy) whereas the image on the right is the
output image (GAN output).

The performance over the entire dataset is then evaluated using the PSNR and
SSIM metrics. The CDF of the two metrics is shown in Fig. 6. The top two
subplots show the CDF curves before applying the GAN model, whereas the bottom
two CDF subplots show the results after applying the GAN model to the images.
The difference is very clear in image quality enhancements as depicted in the
figure e.g., the PSNR values first ranged between 14.5 and 18.0, which after
applying the GAN ranged between 18.5 and 21.0. Similarly, the SSIM range was
0.18-0.25 (before applying GAN) which is improved to 0.53-0.59 (after applying
the GAN model), indicating major improvements in the image quality.

Figure 6: CDF of PSNR (left) and SSIM (right) values of the images in the
training set (top) and the generated images using Pix2Pix GAN (bottom).

The trained Pix2Pix model is then applied to the JHU-Crowd++ dataset to
enhance noisy images in the dataset. The two versions of the JHU-Crowd++
dataset i.e., original (with noisy images) and enhanced (without noisy images)
are then used to train crowd-counting models. The baselines for our analysis
are the crowd models trained on the original dataset. The baseline models are
trained until the loss converges (did not reduce further after 10 epochs).
Then the pretrained models are further trained (finetuned) on the enhanced
dataset (containing images generated with the GAN model). The counting
performance is evaluated using the mean absolute error (MAE) metric in Eq. 5
for both methods i.e., models trained on the original dataset (Baselines), and
models trained on the enhanced dataset (ours). The performance of our proposed
method is compared against all baselines )i.e., MCNN, CMTL, CSRNet, TEDnet,
and SANet) and the results are reported in Fig. 7. The figure shows that the
MAE values for all models using our proposed method are lower (lower MAE means
better counting accuracy) than the baselines (without image denoising).

Figure 7: Performance comparison of various crowd counting models using Mean
Absolute Error (MAE) metric.

##  V Conclusion

In this paper, we propose a two-stage crowd density estimation and counting
model for harsh weather conditions using generative adversarial networks. A
Pix2Pix GAN is applied to create enhanced and clear versions of the blurry and
noisy crowd images taken in harsh weather such as fog, rain, and dust causing
distortions. The enhanced images are then passed to the crowd model for
density estimation. Experiments conducted on the JHU-Crowd dataset using
several mainstream crowd models show superior performance as compared to the
baseline over several metrics. It is worth noting that during the inference,
the proposed method incurs extra delay and thus it may not be suitable for
real-time performances. However, it can be a good choice for applications
requiring high reliability and accuracy.

## References

  * [1] M. Li, Z. Zhang, K. Huang, and T. Tan, “Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection,” in 2008 19th International Conference on Pattern Recognition, pp. 1–4, 2008.
  * [2] I. S. Topkaya, H. Erdogan, and F. Porikli, “Counting people by clustering person detector outputs,” in 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 313–318, 2014.
  * [3] K. Chen, C. C. Loy, S. Gong, and T. Xiang, “Feature mining for localised crowd counting,” in BMVC, 2012.
  * [4] A. B. Chan and N. Vasconcelos, “Bayesian poisson regression for crowd counting,” in 2009 IEEE 12th International Conference on Computer Vision, pp. 545–551, 2009.
  * [5] M. A. Khan, H. Menouar, and R. Hamila, “Revisiting crowd counting: State-of-the-art, trends, and future perspectives,” Image Vision Comput., vol. 129, jan 2023.
  * [6] C. Zhang, H. Li, X. Wang, and X. Yang, “Cross-scene crowd counting via deep convolutional neural networks,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 833–841, 2015.
  * [7] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma, “Single-image crowd counting via multi-column convolutional neural network,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 589–597, 2016.
  * [8] V. A. Sindagi and V. M. Patel, “Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting,” in 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), (Los Alamitos, CA, USA), pp. 1–6, IEEE Computer Society, sep 2017.
  * [9] Y. Li, X. Zhang, and D. Chen, “Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1091–1100, 2018.
  * [10] X. Jiang, Z. Xiao, B. Zhang, X. Zhen, X. Cao, D. S. Doermann, and L. Shao, “Crowd counting and density estimation by trellis encoder-decoder networks,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6126–6135, 2019.
  * [11] X. Cao, Z. Wang, Y. Zhao, and F. Su, “Scale aggregation network for accurate and efficient crowd counting,” in ECCV, 2018.
  * [12] Q. Song, C. Wang, Y. Wang, Y. Tai, C. Wang, J. Li, J. Wu, and J. Ma, “To choose or to fuse? scale selection for crowd counting,” in AAAI, 2021.
  * [13] C. Gao, P. Wang, and Y. Gao, “Mobilecount: An efficient encoder-decoder framework for real-time crowd counting,” in Pattern Recognition and Computer Vision: Second Chinese Conference, PRCV 2019, Xi’an, China, November 8–11, 2019, Proceedings, Part II, p. 582–595, Springer-Verlag, 2019\.
  * [14] M. A. Khan, H. Menouar, and R. Hamila, “Lcdnet: a lightweight crowd density estimation model for real-time video surveillance,” Journal of Real-Time Image Processing, vol. 20, 2023.
  * [15] M. A. Khan, H. Menouar, and R. Hamila, “Dronenet: Crowd density estimation using self-onns for drones,” in 2023 IEEE 20th Consumer Communications & Networking Conference (CCNC), pp. 455–460, 2023.
  * [16] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in NIPS, 2014.
  * [17] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976, 2016.
  * [18] D. C. de Lima, D. Saqui, S. A. T. Mpinda, and J. H. Saito, “Pix2pix network to estimate agricultural near infrared images from rgb data,” Canadian Journal of Remote Sensing, vol. 48, pp. 299 – 315, 2022.
  * [19] R. Toda, A. Teramoto, M. Kondo, K. Imaizumi, K. Saito, and H. Fujita, “Lung cancer ct image generation from a free-form sketch using style-based pix2pix for data augmentation,” Scientific Reports, vol. 12, 2022.
  * [20] A. A. Aljohani and N. R. Alharbe, “Generating synthetic images for healthcare with novel deep pix2pix gan,” Electronics, 2022.
  * [21] L. Zeng, X. Xu, B. Cai, S. Qiu, and T. Zhang, “Multi-scale convolutional neural networks for crowd counting,” 2017 IEEE International Conference on Image Processing (ICIP), pp. 465–469, 2017.
  * [22] L. Boominathan, S. S. S. Kruthiventi, and R. V. Babu, “Crowdnet: A deep convolutional network for dense crowd counting,” Proceedings of the 24th ACM international conference on Multimedia, 2016.
  * [23] W. Liu, M. Salzmann, and P. V. Fua, “Context-aware crowd counting,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5094–5103, 2019.
  * [24] S. Aich and I. Stavness, “Global sum pooling: A generalization trick for object counting with small datasets of large images,” arXiv preprint arXiv:1805.11123, 2018.
  * [25] Z. Chen, J. Cheng, Y. Yuan, D. Liao, Y. Li, and J. Lv, “Deep density-aware count regressor,” in ECAI, 2020.
  * [26] Q. Wang and T. Breckon, “Crowd counting via segmentation guided attention networks and curriculum loss,” IEEE Transactions on Intelligent Transportation Systems, 2022.
  * [27] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” ArXiv, vol. abs/1411.1784, 2014.
  * [28] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * [29] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251, 2017.
  * [30] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 43, pp. 4217–4228, dec 2021.
  * [31] M. S. K. Raghavendra and P. N. Sarappadi, “Transfer learning with pix2pix gan for generating realistic photographs from viewed sketch arts,” Journal of Southwest Jiaotong University, 2022.
  * [32] Y. Qu, Y. Chen, J. Huang, and Y. Xie, “Enhanced pix2pix dehazing network,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8152–8160, 2019.
  * [33] T. Shimada, H. Nishikawa, X. Kong, and H. Tomiyama, “Pix2pix-based monocular depth estimation for drones with optical flow on airsim,” Sensors (Basel, Switzerland), vol. 22, 2022.
  * [34] A. Raposo, A. Azeitona, M. Afonso, and J. M. Sanches, “Ultrasound denoising using the pix2pix gan,”
  * [35] H. an Li, M. Zhang, Z. Yu, Z. Li, and N. Li, “An improved pix2pix model based on gabor filter for robust color image rendering.,” Mathematical biosciences and engineering : MBE, vol. 19 1, pp. 86–101, 2021.
  * [36] P. S. Jadhav, M. Sawal, A. Zagade, P. Kamble, and P. Deshpande, “Pix2pix generative adversarial network with resnet for document image denoising,” 2022 4th International Conference on Inventive Research in Computing Applications (ICIRCA), pp. 1489–1494, 2022.
  * [37] Y. Jiang, Y. Zhang, C. Luo, P. Yang, J. Wang, X. Liang, W. Zhao, R. Li, and T. Niu, “A generalized image quality improvement strategy of cone-beam ct using multiple spectral ct labels in pix2pix gan,” Physics in Medicine & Biology, vol. 67, 2022.
  * [38] H. S. Park, J. Baek, S. K. You, J. K. Choi, and J. K. Seo, “Unpaired image denoising using a generative adversarial network in x-ray ct,” IEEE Access, vol. 7, pp. 110414–110425, 2019.
  * [39] A. Kalpana. and A. John, “Pix2pix gan image synthesis to detect electric vehicle license plate,” 2022 IEEE 4th International Conference on Cybernetics, Cognition and Machine Learning Applications (ICCCMLA), pp. 439–443, 2022.
  * [40] J. Sun, Y. Du, C.-Y. Li, T.-H. Wu, B.-H. Yang, and G. S. P. Mok, “Pix2pix generative adversarial network for low dose myocardial perfusion spect denoising.,” Quantitative imaging in medicine and surgery, vol. 12 7, pp. 3539–3555, 2021.
  * [41] V. A. Sindagi, R. Yasarla, and V. M. Patel, “Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method,” Technical Report, 2020.


# Human Preferences and Robot Constraints Aware Shared Control
for Smooth Follower Motion Execution

Qibin Chen1 Yaonan Zhu1∗ Kay Hansel2 Tadayoshi Aoyama1 and Yasuhisa Hasegawa1
1\. Department of Micro-Nano Mechanical Science and Engineering  Nagoya
University
Nagoya  Aichi  464-8603  Japan
2\. Department of Computer Science  Technical University of Darmstadt
Hochschulstr. 10  64289 Darmstadt  Germany

This work has been submitted to the IEEE for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible.∗Corresponding author email<EMAIL_ADDRESS>work
was supported in part by JST Trilateral AI Research, Japan, under Grant
JPMJCR20G8; and in part by JSPS KAKENHI under Grant JP22K14222.

###### Abstract

With the continuous advancement of robot teleoperation technology, shared
control is used to reduce the physical and mental load of the operator in
teleoperation system. This paper proposes an alternating shared control
framework for object grasping that considers both operator’s preferences
through their manual manipulation and the constraints of the follower robot.
The switching between manual mode and automatic mode enables the operator to
intervene the task according to their wishes. The generation of the grasping
pose takes into account the current state of the operator’s hand pose, as well
as the manipulability of the robot. The object grasping experiment indicates
that the use of the proposed grasping pose selection strategy leads to
smoother follower movements when switching from manual mode to automatic mode.

##  1 Introduction

Shared control, which employs automation to support human operation, has been
widely used in teleoperation systems, since it effectively improves
performance while reducing the physical and mental strain on users. Object
grasping is one of the most important teleoperation tasks and is the subject
of this paper. To address this task, previous work [1] has developed a shared
control system and demonstrated that the system improved grasping performance
and reduced operator fatigue. This shared control framework can guide the
operator to an appropriate grasping pose with the best manipulability of the
robot, by dynamically and continuously blending user intention and automation
assistance. This paper aims to address the problem of shared control system
which does not consider human choices. An alternating shared control system is
proposed, which segregates the automation and manual aspects, empowering users
to intervene at any time. On the other hand, conventional continuous shared
control maintains the combination of manual input and automation throughout
the entire process. In Maeda’s work [2], the advantages of alternating shared
control were demonstrated that the alternating way won a higher score for ease
of use in subjective ratings while keeping the performance as the continuous
method. In addition, to achieve a smooth follower motion execution, the target
grasping pose is selected from all candidates by taking into account not only
the manipulability of the robot, but also the preferences of the operator.

##  2  Shared Control System with Human Preferences

###  2.1 Pose Remapping in Shared Control System

The alternating shared control is realized by setting the status switching
trigger on the VR controller, and processing the positional and directional
gaps between the two states. In our system, the pure teleoperation part
establishes a transformation from the VR coordinate system
($\boldsymbol{p}_{hand,0}^{htc}$, $\boldsymbol{p}_{hand,k}^{htc}$, $0$ means
the initial step, $k$ is for the k’th sampling step) to the robot base
coordinate system ($\boldsymbol{p}_{e,k}^{b}$, $\boldsymbol{p}_{e,0}^{b}$), by
calculating the relative position:
$\boldsymbol{p}_{e,k}^{b}=\boldsymbol{p}_{e,0}^{b}+(\boldsymbol{p}_{hand,k}^{htc}-\boldsymbol{p}_{hand,0}^{htc})$.
For the rotational motion mapping, the absolute orientation related to the
robot base frame is used as
$\boldsymbol{R}_{e,k}^{b}=\boldsymbol{R}_{htc}^{b}\cdot\boldsymbol{R}_{hand,k}^{htc}$.
When the status change is triggered, the position remapping is as follows.

$\boldsymbol{p}_{e,0}^{b}=\boldsymbol{p}_{e,k}^{b}$ (1)
$\boldsymbol{p}_{hand,0}^{htc}=\boldsymbol{p}_{hand,k}^{htc}$ (2)

Meanwhile, the new absolute orientation is filtered by the spheical linear
interpolation (SLERP).

$\boldsymbol{R}_{e,k}^{b}=SLERP(\boldsymbol{R}_{e,k-1}^{b},\boldsymbol{R}_{htc}^{b}\cdot\boldsymbol{R}_{hand,k}^{htc},\alpha)$
(3)

where, $\boldsymbol{\alpha}$ is the interpolation parameter between $0$ and
$1$, representing linear placement position from $\boldsymbol{R_{e,k}^{b}}$ to
$\boldsymbol{R}_{htc}^{b}\cdot\boldsymbol{R}_{hand,k}^{htc}$.

###  2.2 Grasping Pose Selection Based on Human Preferences and Robot
Constraints

Figure 1: Target grasping poses with respect to human preferences, and end-
effector motion curves with two selection strategies. In (a)(b), the three-
dimentional markers demonstrate the current pose of gripper and the selected
grasping poses towards different objects. As the user adjusts the end-effector
pose from (a) pose 1 to (b) pose 2, the selected grasping poses of three
objects chang accordingly. In (c), the method that considers both human-
selected end-effector pose and robot manipulability (orange curve) makes the
motion trajectories shorter and smoother than the method which considers only
robot manipulability (blue curve).

In [1], the target grasping poses were detected from multiple directions using
template matching based object point cloud compensation. However, the final
selection of the grasping pose was solely based on the best robot
manipulability from the library of generated poses. As a result, when utilized
in the alternating shared control, the user’s preferences were not taken into
account. It led to a noticeable jump in robot motion when transitioning from
manual mode to automatic mode.

To enable the system to provide automatic assistance based on human
manipulation outcomes, this paper proposes a feasible solution. The solution
follows these steps: Firstly, for each object, 150 reliable generated grasping
poses are stored when the object is not visually occluded by the manipulator.
Secondly, it filters out candidates with the closest positional and
directional distances in sequence. Thirdly, it selects the most robot-operable
target grasping pose among them. The reason is to avoid wrong grasping poses
being included in the grasping pose library. Given the current quaternions of
the end-effector $\boldsymbol{q}_{ee}$ and candidates in the grasping pose
list
$\boldsymbol{q}_{l}=[\boldsymbol{q}_{1},\boldsymbol{q}_{2},…,\boldsymbol{q}_{i}]$,
the system calculates the angular absolute distance for each candidate.

$d_{a,i}=min(\lVert\boldsymbol{q}_{ee}+\boldsymbol{q}_{i}\rVert,\lVert\boldsymbol{q}_{ee}-\boldsymbol{q}_{i}\rVert)$
(4)

The distance is the chord length of the shortest path that connects the two
quaternions. The top 30 grasping poses closest in the orientation are updated
in the candidate list. Sequentially, the linear distance is calculated between
the gripper position $\boldsymbol{p}_{ee}$ and the candidate
$\boldsymbol{p}_{i}$ from the updated list.

$d_{l,i}=\lVert\boldsymbol{p}_{ee}-\boldsymbol{p}_{i}\rVert$ (5)

The top 6 grasping poses with the shortest distances are collected into a new
list. Then, the robot manipulabilities are calculated and the highest result
is choosed as the target grasping pose. The penalized manipulability, which
considers singular configurations $S(\boldsymbol{\theta})$ and joint limits
$L(\boldsymbol{\theta})$ by
$M(\boldsymbol{\theta})=S(\boldsymbol{\theta})L(\boldsymbol{\theta})$, is
presented in [1]. Automatic approach towards the target is achieved through
interpolation from the current pose to the target pose. Hence, the grasping
pose changes according to the user manipulation and the robot constraints
(Fig. 1(a)(b)).

##  3 Experiment and Result

As the improvement of manipulability on the follower side has been evaluated
in our previous work [1], here we check the level of motion smoothness when
the user preferences are carried out in the automatic mode. The object
grasping experiment is conducted in two modes: one considering only the robot
constraints, and the proposed method which considers both the robot
manipulability and the robot current pose. In each mode, the system first
automatically moves to a fixed preparation pose to simulate user operation,
and then completes the grasping task in automatic mode. The Fig. 1(c)
demonstrates the transformation of the robot’s end-effector pose from the
ready pose to the selected grasping pose. It shows that the proposed grasping
pose selection strategy reduces gripper movements, making them smoother, and
achieves a shorter completion time as the execution speed is the same
($0.1m/s$).

##  4 Conclusions

In this paper, we have applied an alternating shared control that enables
human intervention, enhances the comprehensibility of the robot’s motion, and
smoothes the gap between control mode switching. Positional remapping and
directional spherical linear interpolation are employed to realize the
intervention-possible alternating shared control. The proposed grasping pose
selection takes into account both manual operation and robot manipulability.
The two factors are affected by human preferences and robot constraints,
respectively. The result of an object grasping experiment shows that the
shared control system extends the operators’ wishes and makes the follower’s
motion smooth when switching control modes. Future work includes making
different degrees of freedom have different weights in the cost function of
grasping pose selection, and enabling people to fine-tune the final grasping
pose.

## References

  * [1] Y. Zhu, B. Jiang, Q. Chen, T. Aoyama, and Y. Hasegawa, “A shared control framework for enhanced grasping performance in teleoperation,” _IEEE Access_ , 2023.
  * [2] G. Maeda, “Blending primitive policies in shared control for assisted teleoperation,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 9332–9338.


# Wolfies at SemEval-2022 Task 8: Feature extraction pipeline with
transformers for Multi-lingual news article similarity

Nikhil Goel
CS Department
Stony Brook University
Long Island, NY
<EMAIL_ADDRESS>
Ranjith Reddy
CS Department
Stony Brook University
Long Island, NY
<EMAIL_ADDRESS>

###### Abstract

This work is about finding the similarity between a pair of news articles.
There are seven different objective similarity metrics provided in the dataset
for each pair and the news articles are in multiple different languages. On
top of the pre-trained embedding model, we calculated cosine similarity for
baseline results and feed-forward neural network was then trained on top of it
to improve the results. We also built separate pipelines for each similarity
metric for feature extraction. We could see significant improvement from
baseline results using feature extraction and feed-forward neural network.

Wolfies at SemEval-2022 Task 8: Feature extraction pipeline with transformers
for Multi-lingual news article similarity

Nikhil Goel CS Department Stony Brook University Long Island, NY
<EMAIL_ADDRESS>Ranjith Reddy CS Department Stony Brook University
Long Island, NY<EMAIL_ADDRESS>

##  1 Introduction

For finding similarity between two documents chen-etal-2022-semeval, the
general approach is to get the embeddings for the documents and use some
similarity metric like cosine similarity to get similarity measures. Such work
has been done in multiple domains like research papers, semantic similarity
(Olizarenko and Radchenko, 2021) (Boom et al., 2015) document similarity
(Ostendorff et al., 2020) (Rushkin, 2020) and for news articles (Watters and
Wang, 2000) (Singh and Singh, 2020) (Blokh and Alexandrov, 2017). There have
been other works to find similarity between documents of multiple languages
(Potthast et al., 2008) as well. The most significant difference here is that
authors have used subjective similarity measures like writing style and
authorship for these works.

We do feature extraction corresponding to each similarity metric via a unique
feature extraction pipeline. For example, for geolocation, we extract the
locations from the text. To make it a domain-specific task, we train a network
on top of the generic similarity pre-trained embedding model.

To test our approach, we use a pre-trained model for sentence similarity based
on S-BERT Reimers and Gurevych (2019a) Devlin et al. (2019). We calculate
cosine similarity Rahutomo et al. (2012) as our baseline on the embedding of
the documents. We train a feed-forward network on top of the pre-trained
embedding to learn about our domain-specific task. We also use a large pre-
trained model, which can take more words as input and compare the results. Our
approaches result in an almost 60% MSE improvement over the baseline model.

Our contributions are mentioned below:

  * •

We implement a strong paraphrase-multilingual-MiniLM baseline and show
significant gains in MSE by adding a feedforward neural network on top to
learn domain specific knowledge.

  * •

We do domain-specific feature extraction for each similarity metric to prevent
model from learning incorrect information. This feature extraction helps
improve the final metrics for the task.

  * •

We do qualitative result analysis and show that the baseline model does not
learn relevant information for location/time/entity for the corresponding
metrics. We observe that after adding feature extraction, the model learns
metric specific information.

##  2 System Description

###  2.1 Feature Extraction

We extract the entities from the text using Stanza Qi et al. (2020). Stanza is
a python natural language analysis library that contains tools for parsing
sentences, recognize named entities etc. Through our experiments we observe
that for similarity measures like geography, entities and time, we just need
to extract those relevant named entities from the text. To do this, we build
seven stanza pipelines for the seven similarity measures. Feature extraction
for each of the metric is described below.

  1. 1.

Geography: We extract the location from the article for each pair. If Stanza
cannot extract the location, we use the entire text as the embedding. We
linearize the location and feed them to the pre-trained model if there are
multiple locations. For example, if there are four locations in the article,
the output of the pre-trained model is [4, embedding size]. To further
linearize the input, we take the mean of all the rows, so the linearized input
dims is [1, embedding size]. We repeat this process for the other news
articles and calculate the cosine similarity between the two articles.

  2. 2.

Entities: We extract all the named entities from the article for each pair. We
linearize the input in the same way as geography, calculating the cosine
similarity. There were no cases in this dataset where stanza was unable to
extract entities, but in case there are, they will be handled similarly to
location.

  3. 3.

Time: We extracted time and date entities from the articles for each pair. If
Stanza cannot extract the time entity, we create an embedding for the entire
text instead of the time. The rest of the processing is similar to Geography
and entities.

  4. 4.

For Narrative, Style, Tone and Overall, we give the entire text as input to
the pre-trained embedding and calculate the similarity based on that. In
general, for this slightly subjective measure, the neural network learns the
representations on its own.

Using the feature extraction defined above, we build our baseline model.
Initially, we were giving entire text as input for location class but were
getting similar results (less overfitted but worse on test data). We
hypothesized that it might be because the model is not learning about specific
location class entities but looking at the overall text. To confirm this
hypothesis, we took two news articles that were outputting highly similar
scores for location and manually changed the location of one article to some
random place. The result was that the model was still outputting the two
articles as quite similar, which showed that the model was not learning
correctly. We extracted just the entities for an objective metric like
location, time, and entities and gave that as input to the model. We are
making sure that the model is not learning the wrong information to get
correct results.

In the future, we can to try to extract the dependency parse of the sentence
and give it as an input to the pre-trained embedding as another step in the
feature extraction task. We hypothesize that for the style and tone metric
giving the dependency parse of sentence as input to the neural network will
improve the results. This is because style and tone of writing is author
specific and depends on the writing style of the author which a dependency
parse of the sentence can capture, but we have not used that approach for this
paper.

###  2.2 Baseline Model

For the baseline model, we used the en-en articles as our dataset. We then do
feature extraction to extract relevant features from the article corresponding
to the similarity metric. After that, we got the embeddings for these pairs
from the pre-trained model for sentence similarity (miniLM-v6) Reimers and
Gurevych (2019a) and calculated the cosine similarity to find similarity
between the articles and calculated the loss for each similarity metric
separately.

#### Limitation

The issue with the approaches done historically is that the similarity metrics
were subjective, like comparing two documents to see if the same author has
written them or not. For such tasks, semantic and syntactic features of the
document play a vital role. For our task, we need to focus more on the
article’s content. For the baseline model that we create, there are two
significant issues. First, we use en-en articles to train the model. To test,
we translate other language articles to English using Google translate. This
approach is error-prone as it compounds the error. Second, if we directly use
the cosine similarity of two embedding from a generic document similarity
model, we do not take advantage of the domain-specific task. For example, the
document similarity in the case of medical documents will be much different in
the case of research documents.

Figure 1: Pipeline for the task. The figure depicts the pipeline for
similarity generation containing preprocessing, stanza feature extraction, and
pre-trained S-BERT model steps. If we remove the feed-forward and directly
calculate the similarity score, that is our baseline model.

##  3 Proposed approaches

###  3.1 Feed-forward network

The first idea is to take advantage of the domain-specific task in this work.
To do this, we train a feedforward model on top of the baseline to learn
document similarity for the news articles domain. To implement this idea, we
concatenated the embedding that we got using the pipeline in Figure 1. This
input is fed through a three-layer neural network with a ReLU activation
function after each layer. The final layer is passed through a softmax layer
to get the similarity score between 0 to 1. This model uses MSE loss between
the output and predicted value as the loss function, and the optimizer used to
train this network is SGD.

###  3.2 Doc S-BERT

Doc S-BERT is a BERT-base document similarity model that is also called bert-
base-nli-mean-tokens. The amount of resources needed to use a base pre-trained
model for BERT is much more, but the idea behind using the large doc
similarity model is due to a shortcoming in the miniLM model. The miniLM model
takes a maximum of 256 words of a document, after which it truncates the rest
of the document. The input text length went up to 1000 words, so we believe
there is a loss of important information if we use the miniLM model. Both Doc
S-BERT and MiniLM models belong to the family of Siamese BERT (Figure 2)
Reimers and Gurevych (2019b) models.

Figure 2: Architecture of S-BERT Reimers and Gurevych (2019b)

###  3.3 Multilingual S-BERT

The idea is to use a multilingual pre-trained model to generate embedding
instead of converting other languages to English to prevent the compounding of
errors. We use a multilingual pre-trained similarity embedding generating
model. Multilingual miniLM is a single model that is pre-trained for multiple
languages. The model is called ’paraphrase-multilingual-MiniLM-L12-v2’. The
output embedding dimension from this is similar to our previous model, and the
constraints are also similar. We hypothesize that if we can remove the
compounding of errors, the multilingual model should give better results than
the baseline.

##  4 Experimental Setup

###  4.1 Evaluation Measures

The dataset rates similarity metric between 1 to 4, where 1 means most
similar, and 4 means most dissimilar. To convert this score to a more usable
metric, we subtracted it from the max (4) and divided it by max-min (4-1). The
score gets normalized between 0 to 1, with 1 being most similar and 0 being
most dissimilar. This score is treated as a regression problem as the scores
are rated continuously between 1 to 4 with small buckets.

To evaluate our model, we use MSE loss as the metric. Since we treat this as a
regression problem, MSE loss is a good way to judge how the model performance.

Mean squared error MSE = $\displaystyle\frac{1}{n}\sum_{t=1}^{n}e_{t}^{2}$

where $e_{t}$ is the difference between predicted and actual value.

We have also set a tolerance value to get the accuracy. We have tried
different values of tolerance to see how our results vary.

If the predicted and actual value difference is less than tolerance, we
predict it as a true label; otherwise, we predict it as a false label. We take
the mean over the dataset to get the accuracy.

###  4.2 Preprocessing

We download news articles as HTML files from the URLs given in the dataset. We
use beautiful soup to extract the headings and body of the text in the news
article and store the metadata in JSON files for each article id. There was
some junk data appended at the end of the news article that we manually
removed in many articles. We removed the stop words from these articles
because our pre-trained model has a word limit in the input document.

###  4.3 Experimental design

We used the dataset provided by the task organizers 111URL of the dataset
https://competitions.codalab.org/competitions/33835#learn_the_details-
timetable.. The train dataset has 4964 pairs of news articles with different
language articles. En-en:1800, de-de:857, de-en:577, es-es: 570, tr-tr: 465,
pl-pl: 349, ar-ar: 274, fr-fr: 72. We split the dataset into train and test
sets with the division as 67:33. We did not use a validation set since the
dataset was tiny.

We used a variation of S-BERT for all four approaches as the embeddings
generation pre-trained model. For baseline and first approach, we use a miniLM
model. We use a large miniLM model for the second approach. For the third
approach, we used a small multilingual miniLM model. The input truncates after
256 words in the first approach (miniLM FFN) and the third
approach(multilingual miniLM). For the second approach (Doc-S-BERT), the input
truncates after 256 words.

The output of the embedding size from the small pre-trained model is (384,1).
The input to the neural network will be two * embedding size since we
concatenate the two article embeddings in our approach, it will be (768,1). We
use a three layer feedforward network with layer sizes as (120, 84, 1). After
each layer, we feed the input through a ReLU activation function. The final
layer is passed through a softmax layer to get the similarity score between 0
to 1. There are seven models for each similarity metric as the input is
different for each.

We train the model using SGD with the learning rate at 0.01 and momentum at
0.9. The loss function we chose to train this model is MSE. We trained the
model for eight epochs (Early stopping) as the input size was small, and the
model started overfitting on the data.

##  5 Result and analysis

Metric |  baseline (cosine-sim) MSE |  miniLM (approach1) MSE |  doc-SBERT MSE |  M-miniLM-cosine-sim MSE |  M-miniLM-MSE
---|---|---|---|---|---
Geography |  0.15 |  0.135 |  0.164 |  0.175 |  0.134
Time |  0.132 |  0.074 |  0.079 |  0.171 |  0.109
Entity |  0.288 |  0.106 |  0.098 |  0.327 |  0.133
Narrative |  0.129 |  0.119 |  0.131 |  0.258 |  0.127
Style |  0.192 |  0.075 |  0.085 |  0.136 |  0.081
Tone |  0.187 |  0.077 |  0.077 |  0.131 |  0.075
Overall |  0.131 |  0.122 |  0.132 |  0.248 |  0.128
Table 1: Results of different approaches

###  5.1 Baseline

For location, the MSE loss that we got is around 0.15 (Table 1). The result is
as expected as we hypothesized that just extracting the location entities
should give us a good match for similarity. If we allow the threshold to be
high (0.5), the accuracy, in this case, is around 0.82. Since its regression,
we need to decide the threshold to calculate a metric like an accuracy. With a
threshold of 0.33, accuracy was around 0.67. A low threshold like 0.2 gives an
accuracy of 0.47.

Similarly, for time similarity, we got the MSE as 0.132 (Table 1) and the
accuracy with 0.5 tolerance as 0.834. Comparatively, for time, the performance
of cosine similarity is good, which is the expected behavior from our initial
thoughts since it is a simple task of extracting time from an article and
checking its similarity.

For entities similarity, we got the MSE as 0.288 (Table 1) and the accuracy
with 0.5 tolerance as 0.507. The results are much worse for entities as
compared to other results. Our initial thought was that cosine similarity
should work with simple tasks like entity similarity. We observed that each
article has more than 20 entities at the least, so doing mean of all entities
makes it lose all necessary information. We might need to think of a more
innovative way of using cosine similarity in this case or use the neural
model.

As can be seen from the table, Overall class and Narrative class show
promising results, which is intuitive as the text-similarity of the entire
article captures the narrative class and the overall class similarity. As
expected, Style class and Tone class do not show that great accuracy as both
style and tone require hidden features like semantic and syntactic parse of
the sentence, which text embedding similarity cannot capture. These results
show the need for neural models to find these hidden features with little help
from feature extraction if required.

###  5.2 miniLM

As we can see clearly from the table (Table 1), using a feed-forward network
on top of the pre-trained embedding to make it a domain-specific task
decreases the loss and improves the results for almost all similarity metrics.
These results confirm our hypothesis for domain-specific knowledge.

Our hypothesis for location class not performing well is that the pre-trained
embedding is for sentence similarity, all location entities should be nearby
in the embedding space for such models, and overall it will be hard for such
models to learn the differences between location classes. If we use a
location-specific pre-trained model, we should get good results.

Also, we can see that the improvement is the most for entities, which is
intuitive. The baseline model was poorly performing because embedding all
entities averaged did not give any helpful information. Training a model on
top of it allows the neural model to learn the differences in entities and
output much better results. The logic is similar for the time similarity
metric as well.

We see significant improvement for style and tone classes as the similarity
score loss decreased from 0.192 and 0.187 to 0.075 and 0.077 (Table 1),
respectively. As mentioned previously, these require semantic and syntactic
information of the text, which a cosine similarity cannot find. The neural
model can learn these differences better and show significant improvement.

Narrative class and Overall class gave good results with baseline as they
depend on the embedding of the entire text, but the neural model can improve
on those results slightly. These results are intuitive as the model does not
learn meaningful additional information to improve those metrics.

###  5.3 Doc S-BERT

Using a large BERT model for sentence similarity did not lead to many
significant improvements in test data (Table 1). The shortcoming of the
initial model was that it truncates the text after 256 words. The reason for
these poor results is that most of the information is present in the headlines
and the first paragraph of news articles, and the rest of the article is a
detailed explanation of these texts. This logic makes sense as to why the
larger model is not showing significant improvement. The objective similarity
metric can be calculated using these words, generally less than 256 words, and
gain insightful results.

###  5.4 Multilingual miniLM

For the multilingual model, we first look at the baseline results, which are
the cosine similarity between the embedding.

In Table 1, the most significant difference we see is between the narrative
class and the overall class. For both these measures, our single language
model performed much better than the multilingual model. We hypothesize that
because of multiple languages, the embedding of this model learns less about
the text itself as much as it learns the semantic and syntactic knowledge of
the words.

This hypothesis can be confirmed by looking at the cosine similarity of style
class and tone class, which depends more on these hidden features. As we can
see, the multilingual model is performing better for these metrics because of
the hypothesis stated above. We were unable to test this hypothesis in great
detail but looking through online resources; intuitively, it makes sense.

Even the narrative and overall classes, which gave terrible results (Table 1)
for cosine similarity, have almost the same accuracy as our English model,
which contained only the English data. Using this model for different language
types and still getting the same results strengthens our idea of using a
multilingual model instead of compounding the errors.

The test loss for geography in multilingual miniLM and miniLM is precisely the
same, which shows that the models have been trained similarly for this metric
(Table 1). Our hypothesis in miniLM for location should also hold for this
model.

The multilingual model is giving poor results for time and entity compared to
our previous model (Table 1), which might be because of the feature extraction
problem for other languages. We used Stanza to extract the features, but the
overall accuracy is much lower for other languages than English. To confirm
this hypothesis, we looked at the extracted entities from Other languages, and
the number of entities was much lower than they were in English.

###  5.5 Best model final results

We experiment with multiple approaches using different parameters. Overall,
the multilingual model gave us the most consistent results across all
languages. We choose this to be our best model. Both models performed well in
a few languages but poorly in others. Hence, we chose the most consistent
model across all languages as there can be unseen instances of language in the
test dataset.

We calculated the Pearson correlation coefficient on the test data to further
test this model, which is the official task metric. On the test data 222URL of
the dataset
https://competitions.codalab.org/competitions/33835#learn_the_details-
timetable. that the task organizers have provided, the coefficient was 0.288
for the multilingual model. The eval data has 4902 pairs of news articles and
ten languages.

One of the biggest reasons for the low coefficient was that both the stanza
and multilingual model did not support pl and tr languages from the dataset.
Further, the multilingual model did not support a few other languages from the
eval dataset, which were absent in the training dataset. Due to these two
significant shortcomings, our model performed poorly on a few dataset
instances while performing well on other languages. In the future, we can try
to find other parsers for Named Entity Recognition and different multilingual
models which supports all languages.

###  5.6 Code

The google drive link for the code for this project is given here: link for
google drive The link contains all the generated files, readme for
instructions, and code.

##  6 Conclusions

We created an end-to-end pipeline to find similarities between two news
articles in this work and used multiple approaches to find the most optimal
way to calculate similarity. We were able to deal with the problem of
multilingualism best using a multilingual S-BERT model. We were able to
identify the shortcomings of the pre-trained model for similarity in the
location metric. We also explored using a large model, which did not
significantly improve the results. On the eval data, the multilingual model
showed decent results even on unseen languages, which shows that we can extend
the model to other languages with minor changes.

##  7 Acknowledgments

We did this work as part of the NLP course’s final project, and we would like
to thank prof. Niranjan and the TAs for guiding us in this work.

## References

  * Blokh and Alexandrov (2017) Ilya Blokh and Vassil Alexandrov. 2017.  News clustering based on similarity analysis.  _Procedia Computer Science_ , 122:715–719.  5th International Conference on Information Technology and Quantitative Management, ITQM 2017.
  * Boom et al. (2015) Cedric De Boom, Steven Van Canneyt, Steven Bohez, Thomas Demeester, and Bart Dhoedt. 2015.  Learning semantic similarity for very short texts.  _CoRR_ , abs/1512.00765.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805.
  * Olizarenko and Radchenko (2021) Serhii Olizarenko and Viacheslav Radchenko. 2021.  Method for determining the semantic similarity of arbitrary length texts using the transformers models.
  * Ostendorff et al. (2020) Malte Ostendorff, Terry Ruas, Till Blume, Bela Gipp, and Georg Rehm. 2020.  Aspect-based document similarity for research papers.  _arXiv preprint arXiv:2010.06395_.
  * Potthast et al. (2008) Martin Potthast, Benno Stein, and Maik Anderka. 2008.  A wikipedia-based multilingual retrieval model.  In _European conference on information retrieval_ , pages 522–530. Springer.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A python natural language processing toolkit for many human languages.  _CoRR_ , abs/2003.07082.
  * Rahutomo et al. (2012) Faisal Rahutomo, Teruaki Kitasuka, and Masayoshi Aritsugi. 2012.  Semantic cosine similarity.  In _The 7th International Student Conference on Advanced Science and Technology ICAST_ , volume 4, page 1.
  * Reimers and Gurevych (2019a) Nils Reimers and Iryna Gurevych. 2019a.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Reimers and Gurevych (2019b) Nils Reimers and Iryna Gurevych. 2019b.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Rushkin (2020) Ilia Rushkin. 2020.  Document similarity from vector space densities.  _CoRR_ , abs/2009.00672.
  * Singh and Singh (2020) Ritika Singh and Satwinder Singh. 2020.  Text similarity measures in news articles by vector space model using nlp.  _Journal of The Institution of Engineers (India): Series B_ , 102:329–338.
  * Watters and Wang (2000) Carolyn Watters and Hong Wang. 2000.  Rating news documents for similarity.  _Journal of the American Society for Information Science_ , 51(9):793–804.


# The critical weighted inequalities of the spherical maximal function

Juyoung Lee  Research Institute of Mathematics, Seoul National University,
Seoul 08826, Republic of Korea<EMAIL_ADDRESS>
###### Abstract.

Weighted inequality on the Hardy-Littlewood maximal function is completely
understood while it is not well understood for the spherical maximal function.
For the power weight $|x|^{\alpha}$, it is known that the spherical maximal
operator on ${\mathbb{R}}^{d}$ is bounded on $L^{p}(|x|^{\alpha})$ only if
$1-d\leq\alpha<(d-1)(p-1)-d$ and under this condition, it is known to be
bounded except $\alpha=1-d$. In this paper, we prove the case of the critical
order, $\alpha=1-d$.

###### Key words and phrases:

Spheircal maximal function, weighted inequality

###### 2020 Mathematics Subject Classification:

Primary 42B25, Secondary 35S30

##  1\. Introduction

Let $d\geq 2$ and $d\sigma$ be the normalized Lebesgue measure on
$\mathbb{S}^{d-1}$. Then, for $t>0$, we define the spherical average of $f$ by

$A_{t}f(x)=\int_{\mathbb{S}^{d-1}}f(x-ty)d\sigma(y).$

It is very clear that $A_{t}$ is a bounded operator on $L^{p}$ for any $1\leq
p\leq\infty$. When we take the supremum on $t>0$, we get the following
(global) spherical maximal function:

$Mf(x)=\sup_{t>0}|A_{t}f(x)|.$

Stein [16] proved that $M$ is bounded on $L^{p}$ if and only if $p>d/(d-1)$
when $d\geq 3$. Later, Bourgain [2] obtained that $M$ is bounded on $L^{p}$ if
and only if $p>2$ when $d=2$. A few years later, Mockenhaupt, Seeger, and
Sogge [11] proved the same result using the local smoothing estimate.

So far, many researches have been devoted to studying the spherical maximal
function in various settings. For example, we can define the local spherical
maximal function as follows:

$M_{c}f(x)=\sup_{1<t<2}|A_{t}f(x)|.$

We have so called $L^{p}$-improving phenomenon for $M_{c}$, which means that
$M_{c}$ is bounded from $L^{p}$ to $L^{q}$ for some $p<q$. Schlag, Sogge [15]
obtained the sharp range of $p,q$ except for some boundary points, and S. Lee
[10] obtained the result on the boundary lines except the case
$(p,q)=(5/2,5)$.

In this paper, we are interested in the weighted inequalities of $M$. This was
first considered by Duoandikoetxea, Vega [5]. They proved that $M$ is bounded
on $L^{p}(|x|^{\alpha})$ if $p>d/(d-1)$ and $1-d<\alpha<(d-1)(p-1)-1$. This
result is sharp except the critical case, $\alpha=1-d$. For the critical
weight $|x|^{1-d}$, it is known that $M$ is bounded on
$L^{p}_{rad}(|x|^{1-d})$ when $1<p\leq\infty$ for $d\geq 3$, and when
$2<p\leq\infty$ for $d=2$ (see [4]). Here, $L^{p}_{rad}(|x|^{1-d})$ is the
subspace of $L^{p}(|x|^{1-d})$ consists of radial functions. Recently, Nowak,
Roncal, Szarek [12] obtained sharp conditions for the spherical mean Radon
transform on radial functions to be bounded on weighted spaces. Meanwhile,
Lacey [8] proved that sparse bounds on the spherical maximal function implies
some weighted inequalities. In [8], the author tried to characterize the class
of weight which makes the spherical maximal function bounded, while it is
mentioned that $A_{p}$-weight is not a correct tool to characterize such
weight (see also [3]). The bilinear weighted inequalities of the spherical
maximal function is also considered by Roncal, Shrivastava, Shuin [13].

The following is the main theorem which says that $M$ is bouned on the
critical weighted space $L^{p}(|x|^{1-d})$.

######  Theorem 1.1.

  1. (1)

Let $d=2$ Then, $M$ is bounded on $L^{p}(|x|^{-1})$ if and only if $p>2$.

  2. (2)

Let $d\geq 3$. Then, $M$ is bounded on $L^{p}(|x|^{1-d})$ when $p\geq 2$.

We believe that when $d\geq 3$, $M$ is also bounded on $L^{p}(|x|^{1-d})$ when
$d/(d-1)<p<2$. To prove Theorem 1.1, we first consider $M_{c}$ and decompose
it using the Littlewood-Paley decomposition. To recover the estimate for $M$,
we use a modification of the standard argument (see, for example, [2], [14]).
For the purpose, we define the Littlewood-Paley decomposition. Let $\chi$ be a
smooth function such that
$\operatorname{supp}\chi\subset(1-10^{-2},2+10^{-2})$ and
$1=\sum_{j\in{\mathbb{Z}}}\chi(s/2^{j})$ for all $s\neq 0$. We define the
projection $\mathcal{P}_{j}$ by
$\widehat{\mathcal{P}_{j}f}(\xi)=\widehat{f}(\xi)\chi_{j}(|\xi|)$ where
$\chi_{j}(x)=\chi(s/2^{j})$. Then, we define

$M_{j}f(x)=M_{c}\mathcal{P}_{j}f(x).$

Also, we denote $\tilde{\chi}:{\mathbb{R}}^{d}\to{\mathbb{R}}$ by a smooth
function which is supported on $\\{x:1/100<|x|<100\\}$ and equals to 1 when
$1/10<|x|<10$, and then $\tilde{\chi}_{j}(x)=\tilde{\chi}(x/2^{j})$. We obtain
sharp estimates of $M_{j}$ as follows.

######  Proposition 1.2.

Let $d\geq 2$, $j,k\geq 0$, and $d/(d-1)<p<\infty$. Then, for
$\delta=\delta(d,p)>0$, we have

(1.1) $\displaystyle\|\chi_{-k}(|\cdot|)M_{j}f\|_{L^{p}(|x|^{1-d})}^{p}$
$\displaystyle\lesssim
2^{-\delta|j-k|}\|\tilde{\chi}\mathcal{P}_{j}f\|_{L^{p}(|x|^{1-d})}^{p}+\sum_{m\in{\mathbb{Z}}}2^{-Nj-(d-1)|m|-\delta
k}\|\chi_{m}\mathcal{P}_{j}f\|_{L^{p}(|x|^{1-d})}^{p}$

for any $N>0$.

######  Remark 1.

In the above proposition, the main term is obviously the first term on the
right hand side of (1.1), and the other terms come from the Schwartz tail.
Throughout the paper, we will encounter similar issues several times. Since
those terms have sufficiently nice decay, they are always harmless.
Nevertheless, we rigorously handle all those situations. The reader who
believes that the Schwartz tail is harmless may ignore these parts, precisely,
every term containing $2^{-Nj}$.

We briefly explain the novelty of this paper. Consider a suitable bounded
function $F_{j}$ which is supported in an annulus $\\{x:||x|-1|<2^{-j}\\}$ and
the Fourier transform of $F_{j}$ is supported on $\\{\xi:|\xi|\sim 2^{j}\\}$.
This example says that $\|M_{j}F_{j}\|_{L^{p}(|x|^{1-d})}\gtrsim
2^{-j/p}\approx\|F_{j}\|_{L^{p}(|x|^{1-d})}$ since $|M_{j}F_{j}(x)|\gtrsim 1$
when $|x|\lesssim 2^{-j}$. To prove that the spherical maximal function is
bounded, it is essential that $\|M_{j}f\|_{L^{p}}\lesssim
2^{-cj}\|f\|_{L^{p}}$ for a positive number $c>0$. However, on the weighted
space $L^{p}(|x|^{1-d})$, we cannot expect this exponential decay as we can
see at the above. To overcome this difficulty, we decompose $x$ dyadically.
Proposition 1.2 tells us that the main contribution of $M_{j}$ on
$L^{p}(|x|^{1-d})$ occurs when $|x|\sim 2^{-j}$. This gives us the
orthogonality and allows us to prove our main theorem. The proof of
Proposition 1.2 has two parts, $j>k$ and $j\leq k$. In the former case, we use
the local smoothing estimate of the wave operator. However, since we are
considering the critical weight, $\epsilon$-loss on the order of smoothing is
not allowed. When $d\geq 4$, we have the local smoothing estimates without the
$\epsilon$-loss (see [7]) while we do not have such estimates when $d=2,3$.
However, when we consider $L^{p}$–$L^{q}$ estimate, we have the critical local
smoothing estimate. Thus, we use the $L^{p}$–$L^{q}$ local smoothing estimates
and Hölder’s inequality to recover the $L^{p}$-estimate. Since $j>k$, the
singularity of $|x|^{1-d}$ is harmless. In the latter case $j\leq k$, the
blow-up of $|x|^{1-d}$ is no more allowable, so we handle this case using the
spherical harmonic expansion. The key idea is that we handled the
$t$-integration of the wave operator $\mathcal{W}_{t}$ using the Plancherel
theorem, as a Fourier transform of $|\xi|$, the modulus of the frequency
variable.

### Notations

  1. (1)

We denote $A\lesssim B$ when there is an implicit constant $C>0$ which is
independent of $A$ and $B$. In the context, $C$ may depend on $p,d,N$.
Sometimes, we explicitly denote the implicit constant $C_{a,b}$ which depends
on $a,b$.

  2. (2)

We abuse several notations for the $L^{p}$-spaces. $L^{p}(w)$ denotes the
$L^{p}$-space on a suitable domain with the weight $w$. The domain will depend
on the context, but it will be clear. Also, we denote
$L^{p}_{x,t}({\mathbb{R}}^{d}\times I)$ by the $L^{p}$-space with variable
$(x,t)\in{\mathbb{R}}^{d}\times I$. Finally, we also denote $L^{p}_{x}(w;D)$
by the $L^{p}$-space with the variable $x$, the domain $D$ and the weight $w$.

### Acknowledgement

The author would like to thank Kalachand Shuin for a careful reading of the
first version of the draft and finding some typos.

##  2\. Preliminaries

###  2.1. Basic properties

We first review some basic properties which are widely used in various
literature. For a rectangle $R$ in ${\mathbb{R}}^{d}$, we denote $R^{\ast}$ by
the dual rectangle of ${\mathbb{R}}^{d}$ centered at the origin. Then, we can
find a smooth function $\varphi_{R}$ such that $|\varphi_{R}(x)|\sim 1$ on
$R$, rapidly decreasing outside of $R$, and $\widehat{\varphi_{R}}$ is
supported on $R^{\ast}$. For example, when $R$ is a cube centered at the
origin with sidelength $r$, then $\varphi_{R}$ satisfies

$|\varphi_{R}(x)|\lesssim(1+\frac{|x|}{r})^{-N}.$

Then, we have the local orthogonality on $R$.

######  Lemma 2.1.

For a rectangle $R$ and a family of functions $\\{h_{i}\\}$, we assume that
$\\{\operatorname{supp}\widehat{h_{i}}+R^{\ast}\\}$ is a family of finitely
overlapping annulus of the form $\\{\xi:|\xi|\sim 2^{i}\\}$. Then we have

$\int_{R}\sum_{i}|h_{i}(x)|^{p}dx\lesssim\int|\sum_{i}h_{i}(x)\varphi_{R}(x)|^{p}dx$

when $p\geq 2$.

This lemma is straightforward from $|\varphi_{R}(x)|\sim 1$ and the
Littlewood-Paley theory. The following lemma tells us that we may consider the
averaging operator to control the maximal function.

######  Lemma 2.2 ([10], see also [9]).

Let $1\leq p\leq\infty$ and $F:{\mathbb{R}}^{d}\times[1,2]\to{\mathbb{R}}$ be
a smooth function. Then, for any $\lambda>0$ and $x\in{\mathbb{R}}^{d}$, we
have

$\sup_{t\in[1,2]}|F(x,t)|\lesssim\lambda^{\frac{1}{p}}\|F(x,\cdot)\|_{L^{p}([1,2])}+\lambda^{-\frac{1}{p^{\prime}}}\|\partial_{t}F(x,\cdot)\|_{L^{p}([1,2])}.$

Next, we introduce the asymptotic expansion of $\widehat{d\sigma}$ using the
asymptotic formula of the Bessel function (see, for example, [17]):

(2.1) $\textstyle\widehat{d\sigma}(\xi)=\sum_{\pm,\,0\leq k\leq
N}\,C_{j}^{\pm}|\xi|^{-\frac{d-1}{2}-k}e^{\pm
i|\xi|}+E_{N}(|\xi|),\quad|\xi|\geq 1,$

where $E_{N}$ is a smooth function satisfying $|({d}/{dr})^{l}E_{N}(r)|\leq
Cr^{-l-\frac{N+1}{4}},$ $0\leq l\leq(N+1)/{4}$ for $r\geq 1$ and a constant
$C>0$. By the Fourier inversion, we have

$A_{t}\mathcal{P}_{j}f(x)=\int
e^{ix\cdot\xi}\chi_{j}(|\xi|)\widehat{f}(\xi)\widehat{d\sigma}(t\xi)d\xi.$

By fixing sufficiently large $N>0$, the contribution from $E_{N}$ is very
small. Also, from (2.1), it suffices to handle the contribution from
$|\xi|^{-(d-1)/2}e^{i|\xi|}$ since the others are similarly, and easily
handled.

###  2.2. Local smoothing estimates for the wave operator

The local smoothing estimate is one of the common tool in the study of maximal
function. The wave operator defined by

$\mathcal{W}_{t}f(x)=\int e^{i(x\cdot\xi+t|\xi|)}\widehat{f}(\xi)d\xi$

is deeply related to the spherical maximal function. As we have seen in
Section 2.1, the main contribution of $A_{t}\mathcal{P}_{j}$ comes from
$\mathcal{W}_{t}[|D|^{-1/2}\mathcal{P}_{j}f]$. We denote

${\mathbb{A}}_{\lambda}=\\{\eta\in{\mathbb{R}}^{2}:2^{-1}\lambda\leq|\eta|\leq
2\lambda\\}$

for $\lambda>0$. The sharp $L^{p}$-estimates of $\mathcal{W}$ is obtained by
Guth, Wang, and Zhang [6]. By interpolation with the trivial
$L^{1}$-$L^{\infty}$ estimate, we get the following sharp $L^{p}$-$L^{q}$
estimates with $\epsilon$-loss.

######  Proposition 2.3 ([6], see also [15], [9]).

Let $2\leq p\leq q$, $1/{p}+3/{q}\leq 1$, and $\lambda\geq 1$. Then, the
estimate

(2.2)
$\big{\|}\mathcal{W}_{t}g\big{\|}_{L^{q}_{x,t}({\mathbb{R}}^{2}\times\mathbb{[}1,2]])}\leq
C_{\epsilon,p,q}\lambda^{(\frac{1}{2}+\frac{1}{p}-\frac{3}{q})+\epsilon}\|g\|_{L^{p}}$

holds for a constant $C_{\epsilon,p,q}>0$ for any $\epsilon>0$ whenever
$\operatorname{supp}\widehat{g}\subset\mathbb{A}_{\lambda}$.

However, our problem concerns the weight of the critical order, so the
$\epsilon$-loss is not allowable. when $d=2$, S. Lee [10] obtained the
critical $L^{p}$-$L^{q}$ local smoothing estimate for some $p<q$ using Wolff’s
sharp bilinear cone restriction estimate (see [19]). Using the same method, we
can obtain the critical $L^{p}$-$L^{q}$ local smoothing estimate for some
$p<q$ when $d\geq 3$ (without $\epsilon$-loss). However, for our purpose, when
$d\geq 3$, estimates from the Strichartz estimate which is also mentioned in
[10] is enough (see (1.10) in [10]).

######  Proposition 2.4 ([10]).

Let $2\leq p\leq q$, $\frac{1}{p}+\frac{d+1}{(d-1)q}\leq 1$, and $\lambda\geq
1$. Also, we assume $q>14/3$ when $d=2$ and $q\geq\frac{2(d+1)}{d-1}$ when
$d\geq 3$. Then, for a constant $C_{p,q,d}>0$, we have the following estimates
for any $g$ such that
$\operatorname{supp}\widehat{g}\subset\mathbb{A}_{\lambda}$:

(2.3) $\|\mathcal{W}_{t}g\|_{L^{q}_{x,t}({\mathbb{R}}^{d}\times[1,2])}\leq
C_{p,q,d}\lambda^{\frac{d-1}{2}+\frac{1}{p}-\frac{d+1}{q}}\|g\|_{L^{p}}.$

When $d\geq 4$, Heo, Nazarov, Seeger [7] obtained the critical local smoothing
estimates of $\mathcal{W}$ for $p>2(d-1)/(d-3)$.

######  Proposition 2.5.

Let $d\geq 4$ and $p>2(d-1)/(d-3)$. Then, we have the following estimates

$\|\mathcal{W}_{t}g\|_{L^{p}_{x,t}({\mathbb{R}}^{d}\times[1,2])}\leq
C_{p,d}\lambda^{\frac{d-1}{2}-\frac{d}{p}}\|g\|_{L^{p}}$

for some constant $C_{p,d}>0$ whenever
$\operatorname{supp}\widehat{g}\subset\mathbb{A}_{\lambda}$.

We finally remark that allowing the $\epsilon$-loss, the sharp local smoothing
estimates beyond the bilinear method is obtained by Beltran and Saari [1].

###  2.3. Spherical harmonics

In this section we review some basic properties of spherical harmonics. We
refer Chapter 4 of [18] for details. Let $\mathcal{H}_{k}$ be the space of
spherical harmonics of degree $k$. This means that any element of
$\mathcal{H}_{k}$ is a restriction of a degree $k$ harmonic polynomial on
$\mathbb{S}^{d-1}$. Then, it is well known that the collection of all finite
linear combinations of elements of $\bigcup_{k=0}^{\infty}\mathcal{H}_{k}$ is
dense in $L^{2}(\mathbb{S}^{d-1})$. We fix an orthonormal basis of
$\mathcal{H}_{k}$ by

$\mathcal{B}_{k}=\\{Y_{i}^{k}:1\leq i\leq d_{k}\\}$

where $d_{k}$ is the dimension of $\mathcal{H}_{k}$. Then,
$\bigcup_{k=0}^{\infty}\mathcal{B}_{k}$ forms an orthonormal basis of
$L^{2}(\mathbb{S}^{d-1})$. Also, for any function $f\in
L^{2}({\mathbb{R}}^{d})$, we may find the unique expansion

(2.4)
$f(x)=\sum_{k=0}^{\infty}\sum_{i=1}^{d_{k}}a_{i}^{k}(r)Y_{i}^{k}(\theta)$

where $x=r\theta$ for $r>0$, $\theta\in\mathbb{S}^{d-1}$. The following is an
interesting property of spherical harmonics which is crucial in our argument.

######  Lemma 2.6 (Corollary 2.9, Lemma 2.18, [18]).

Suppose $f(x)=f_{0}(r)Y(\theta)$ where $x=r\theta$, $r>0$,
$\theta\in\mathbb{S}^{d-1}$, and $Y\in\mathcal{H}_{k}$ for $k\geq 0$. Then, we
have

$\widehat{f}(\xi)=\left\\{\int_{0}^{\infty}f_{0}(r)\phi_{k}(r\rho)r^{d-1}dr\right\\}Y(\omega)$

for a function $\phi_{k}$ where $\xi=\rho\omega$ for $\rho>0$ and
$\omega\in\mathbb{S}^{d-1}$. Also, for a constant $C(d)>0$, we have
$|\phi_{k}(s)|\leq C(d)$ for any $k\geq 0$, $s\in{\mathbb{R}}$.

Indeed, Lemma 2.6 is not stated as above in [18], but it is contained in the
proof. Also, we note that Lemma 2.6 implies

(2.5)
$\int_{\mathbb{S}^{d-1}}Y(\theta)e^{-ir\rho\theta\cdot\omega}d\theta=\phi_{k}(r\rho)Y(\omega)$

for any $Y\in\mathcal{H}_{k}$. We may find the expansion for $\widehat{f}$ as
follows:

(2.6)
$\widehat{f}(\xi)=\sum_{k=0}^{\infty}\sum_{i=1}^{d_{k}}b_{i}^{k}(\rho)Y_{i}^{k}(\omega)$

where $\xi=\rho\omega$ for $\rho>0$, $\omega\in\mathbb{S}^{d-1}$. Then, by the
orthogonality, we see that

$\|f\|_{L^{2}}=(\sum_{k=0}^{\infty}\sum_{i=1}^{d_{k}}\int_{0}^{\infty}|b_{i}^{k}(\rho)|^{2}\rho^{d-1}d\rho)^{\frac{1}{2}}$

holds.

##  3\. Proof of the main theorem

In this section, we prove Theorem 1.1 assuming Proposition 1.2. For
$n\in{\mathbb{Z}}$, we define $\mathcal{P}_{<n}$ by
$(\mathcal{P}_{<n}f)^{\wedge}(\xi)=\sum_{j<n}\chi_{j}(|\xi|)\widehat{f}(\xi)$.
Also, we denote $\chi_{<n}(s)=\sum_{j<n}\chi_{j}(s)$. Before the proof of the
main theorem, we introduce a technical lemma handling Schwartz tails.

######  Lemma 3.1.

Suppose $p>d/(d-1)$. For $k,j\geq 0$ and any $N>0$,

(3.1) $\displaystyle\int_{|x|\sim 2^{k}}|M_{j}f(x)|^{p}|x|^{1-d}dx$
$\displaystyle\lesssim 2^{-\beta j}\int_{|x|\sim
2^{k}}|\mathcal{P}_{j}f(x)|^{p}|x|^{1-d}dx+\sum_{m\in{\mathbb{Z}}}2^{-N(j+k)-(d-1)|m|}\int_{|x|\sim
2^{m}}|\mathcal{P}_{j}f(x)|^{p}|x|^{1-d}dx$

holds for some $\beta=\beta(d,p)>0$.

Heuristically, since $M_{c}$ is a local maximal operator, the contribution of
$M_{j}f$ on the annulus $\mathbb{A}_{2^{k}}$ comes from the values of
$\mathcal{P}_{j}f$ on an $O(1)$-neighborhood of the same annulus. However, the
restriction of $\mathcal{P}_{j}f$ on the annulus is no more localized on the
Fourier side. Therefore, we are obliged to handle the Schwartz tail which is
the second term on the right hand side of (3.1).

###### Proof of lemma 3.1.

As above, we obtain that

$\int_{|x|\sim
2^{k}}|M_{c}[(1-\tilde{\chi}_{k})\mathcal{P}_{j}f](x)|^{p}|x|^{1-d}dx=0.$

Also, from [2] and [16] (see also [14]), we have

(3.2) $\int|M_{j}g(x)|^{p}dx\lesssim 2^{-\alpha
j}\int|\mathcal{P}_{j}g(x)|^{p}dx\lesssim 2^{-\alpha j}\int|g(x)|^{p}dx$

for some $\alpha>0$ depending on $d,p$ when $p>d/(d-1)$. We denote
$\mathcal{P}_{<j-10}=\mathcal{Q}_{j}^{1}$, and $\sum_{j-10\leq j^{\prime}\leq
j+10}\mathcal{P}_{j^{\prime}}=\mathcal{Q}_{j}^{2}$. Then, by the triangle
inequality, we have

(3.3) $\displaystyle\int_{|x|\sim 2^{k}}|M_{j}f(x)|^{p}|x|^{1-d}dx$
$\displaystyle\lesssim
2^{k(1-d)}(\sum_{i=1,2}\|M_{c}[\mathcal{Q}_{j}^{i}\tilde{\chi}_{k}\cdot\mathcal{P}_{j}f]\|_{L^{p}}+\sum_{j^{\prime}>j+10}\|M_{c}[\mathcal{P}_{j^{\prime}}\tilde{\chi}_{k}\cdot\mathcal{P}_{j}f]\|_{L^{p}})^{p}.$

Note that
$(\mathcal{Q}_{j}^{1}\tilde{\chi}_{k}\cdot\mathcal{P}_{j}f)^{\wedge}(\xi)\neq
0$ only if $|\xi|\sim 2^{j}$ while for $j^{\prime}>j+10$,
$(\mathcal{P}_{j^{\prime}}\tilde{\chi}_{k}\cdot\mathcal{P}_{j}f)^{\wedge}(\xi)\neq
0$ only if $|\xi|\sim 2^{j^{\prime}}$. Thus, by (3.2) and the boundedness of
$M_{c}$, we obtain that the left hand side of (3.3) is bounded by the sum of
the following three quantities.

(3.4)
$2^{k(1-d)-\alpha^{\prime}j}\int|\mathcal{Q}_{j}^{1}\tilde{\chi}_{k}\cdot\mathcal{P}_{j}f(x)|^{p}dx,$
(3.5)
$2^{k(1-d)}\int|\mathcal{Q}_{j}^{2}\tilde{\chi}_{k}\cdot\mathcal{P}_{j}f(x)|^{p}dx,$
(3.6)
$2^{k(1-d)-\alpha^{\prime}j^{\prime}}\sum_{j^{\prime}>j+10}\int|\mathcal{P}_{j^{\prime}}\tilde{\chi}_{k}\cdot\mathcal{P}_{j}f(x)|^{p}dx.$

Here, $\alpha^{\prime}>0$ is a suitable number which is smaller than $\alpha$.
From the spirit of the uncertainty principle, $\widehat{\tilde{\chi}_{k}}$ is
essentially supported in a ball of radius $2^{-k}$ centered at the origin.
Thus, we may expect that (3.4) will be the main term and the others are error
terms. We first see (3.4). By definition, we have

$\displaystyle\mathcal{Q}_{j}^{1}\tilde{\chi}_{k}(x)$
$\displaystyle=\int\widehat{\tilde{\chi}_{k}}(\xi)\chi_{<j-10}(\xi)e^{ix\cdot\xi}d\xi$
$\displaystyle=\int\tilde{\chi}_{k}(y)\big{[}\int
e^{i(x-y)\cdot\xi}\chi_{<j-10}(\xi)d\xi\big{]}dy.$

This implies that when $|x|\sim 2^{m}$ for $m\in{\mathbb{Z}}$,

(3.7)
$|\mathcal{Q}_{j}^{1}\tilde{\chi}_{k}(x)|\lesssim\begin{cases}2^{-N(j+max\\{m.k\\})},&|m-k|\geq
10,\\\ 1&|m-k|<10\end{cases}$

holds for any $N>0$. Thus, (3.4) is bounded by the right hand side of (3.1) as
desired.

Next, we see (3.5). Similarly as before, we obtain that when $|x|\sim 2^{m}$
for $m\in{\mathbb{Z}}$,

(3.8)
$|\mathcal{Q}_{j}^{2}\tilde{\chi}_{k}(x)|\lesssim\begin{cases}2^{-N(j+max\\{m.k\\})},&|m-k|\geq
10,\\\ 2^{-N(j+k)}&|m-k|<10\end{cases}$

holds for any $N>0$. Again, this implies (3.5) is bounded by the right hand
side of (3.1), as desired.

Finally, we see (3.6). The proof is almost similar as before. By the same
argument, we obtain that when $|x|\sim 2^{m}$ for $m\in{\mathbb{Z}}$,

(3.9)
$|\mathcal{P}_{j^{\prime}}\tilde{\chi}_{k}(x)|\lesssim\begin{cases}2^{-N(j^{\prime}+\max\\{m,k\\})},&|m-k|\geq
10,\\\ 2^{-N(j^{\prime}+k)},&|m-k|<10\end{cases}$

holds for any $N>0$. Since $j^{\prime}>j+10$, by taking summation in
$j^{\prime}$, we obtain that (3.6) is bounded by the right hand side of (3.1),
which completes the proof. ∎

###### Proof of Theorem 1.1.

By a scaling argument, it suffices to consider

$\tilde{M}f(x)=\sup_{0<t<1}|A_{t}f(x)|.$

For each $n\geq 0$, we can easily check that

$\sup_{t\sim 2^{-n}}|A_{t}\mathcal{P}_{<n}f(x)|\lesssim M_{HL}f(x)$

holds independent of $n$, where $M_{HL}$ is the Hardy-Littlewood maximal
operator on ${\mathbb{R}}^{d}$. Therefore, we have

(3.10) $\displaystyle\int|\tilde{M}f(x)|^{p}|x|^{1-d}dx$
$\displaystyle\lesssim\int\big{(}\sup_{n\geq 0}\sup_{t\sim
2^{-n}}|A_{t}\mathcal{P}_{<n}f(x)|^{p}+\sup_{n\geq 0}\sup_{t\sim
2^{-n}}|\sum_{j\geq n}A_{t}\mathcal{P}_{j}f(x)|^{p}\big{)}|x|^{1-d}dx$
$\displaystyle\lesssim\int|M_{HL}f(x)|^{p}|x|^{1-d}dx+\sum_{k\in{\mathbb{Z}},n\geq
0}\int_{|x|\sim 2^{-k}}\sup_{t\sim 2^{-n}}|\sum_{j\geq
n}A_{t}\mathcal{P}_{j}f(x)|^{p}|x|^{1-d}dx.$

It is well known that

$\int|M_{HL}f(x)|^{p}|x|^{1-d}dx\lesssim\int|f(x)|^{p}|x|^{1-d}dx$

since $|x|^{1-d}$ is an $A_{p}$-weight. Thus, we only consider the second part
of the right handed side of (3.10).

For simplicity, we denote $f_{n}(x)=f(x/2^{n})$ for a function $f$. Notice
that $[\mathcal{P}_{j}f]_{n}=\mathcal{P}_{j-n}f_{n}$. Then, we have

$\displaystyle\int_{|x|\sim 2^{-k}}\sup_{t\sim 2^{-n}}|\sum_{j\geq
n}A_{t}\mathcal{P}_{j}f(x)|^{p}|x|^{1-d}dx=2^{-n}\int_{|x|\sim
2^{-k+n}}|\sum_{j\geq n}M_{j-n}f_{n}(x)|^{p}|x|^{1-d}dx.$

Now there are two cases, $k<n$ and $k\geq n$. We first handle the case $k<n$.
In this case, we have

(3.11) $\displaystyle\sum_{n\geq 0}\sum_{k<n}2^{-n}\int_{|x|\sim
2^{-k+n}}|\sum_{j\geq n}M_{j-n}f_{n}(x)|^{p}|x|^{1-d}dx$
$\displaystyle=\sum_{n\geq 0}2^{-n}\sum_{k>0}\int_{|x|\sim 2^{k}}|\sum_{j\geq
0}M_{j}f_{n}(x)|^{p}|x|^{1-d}dx$ $\displaystyle\leq\sum_{n\geq
0}2^{-n}\sum_{k>0}\big{(}\sum_{j\geq 0}(\int_{|x|\sim
2^{k}}|M_{j}f_{n}(x)|^{p}|x|^{1-d}dx)^{\frac{1}{p}}\big{)}^{p}$

by the triangle inequality. By Lemma 3.1,
$[\mathcal{P}_{j}f]_{n}=\mathcal{P}_{j-n}f_{n}$, and Höler’s inequality, we
have

$\displaystyle\eqref{k<n case}\lesssim$ $\displaystyle\sum_{n,k,j\geq
0}2^{-\beta^{\prime}j}\int_{|x|\sim
2^{k-n}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx$ $\displaystyle+\sum_{n,k,j\geq
0,\,m\in{\mathbb{Z}}}2^{-N(j+k)-(d-1)|m|}\int_{|x|\sim
2^{m-n}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx$ $\displaystyle\eqqcolon$
$\displaystyle I_{1}+I_{2}$

where $\beta^{\prime}>0$ is a suitable number smaller than $\beta$ in Lemma
3.1.

By changing summations, we get

$I_{1}=\sum_{j\geq 0,k\in{\mathbb{Z}}}2^{-\beta^{\prime}j}\int_{|x|\sim
2^{k}}\sum_{n\geq\max\\{0,-k\\}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx.$

By covering $\\{x:|x|\sim 2^{k}\\}$ by cubes with sidelength $\sim 2^{k}$
which are finitely overlapping and contained in a bigger annulus $\\{x:|x|\sim
2^{k}\\}$, we obtain

$\int_{|x|\sim
2^{k}}\sum_{n\geq\max\\{0,-k\\}}|\mathcal{P}_{j+n}f(x)|^{p}dx\lesssim\int|\mathcal{P}_{>\max\\{0,-k\\}}f(x)|^{p}(1+2^{-k}|x|)^{-N}dx$

by Lemma 2.1, for a sufficiently large $N>0$. We claim that

(3.12)
$\int|\mathcal{P}_{>-k}f(x)|^{p}(1+2^{-k}|x|)^{-N}dx\lesssim\int|f(x)|^{p}(1+2^{-k}|x|)^{-N}dx$

for any $k\in{\mathbb{Z}}$. Indeed, since $\mathcal{P}_{<-k}f=f\ast\psi_{k}$
where $|\psi_{k}(x)|\lesssim 2^{-dk}(1+2^{-k}|x|)^{-N}$ for any $N>0$, (3.12)
comes from the following simple inequality:

$2^{-dk}\int(1+2^{-k}|x-y|)^{-N}(1+2^{-k}|x|)^{-N}dx\lesssim(1+2^{-k}|y|)^{-N}.$

It suffices to check this for $k=0$ by scaling, and it is straightforward.
Therefore,

(3.13) $\displaystyle I_{1}$ $\displaystyle\lesssim\sum_{j\geq
0}2^{-\beta^{\prime}j}\int(|\mathcal{P}_{>0}f(x)|^{p}+|f(x)|^{p})\sum_{k\in{\mathbb{Z}}}2^{k(1-d)}(1+2^{-k}|x|)^{-N}dx$
$\displaystyle\lesssim\int(|\mathcal{P}_{>0}f(x)|^{p}+|f(x)|^{p})|x|^{1-d}dx$
$\displaystyle\lesssim\int|f(x)|^{p}|x|^{1-d}dx$

as desired, since $|x|^{1-d}$ is an $A_{p}$-weight. For the later use, we
remark the following inequality:

(3.14) $\sum_{n\in{\mathbb{Z}}}2^{n(d-1)}(1+2^{n}|x|)^{-N}\lesssim|x|^{1-d}.$

Similarly, we now control $I_{2}$. Since

$\sum_{n,k,j,m\geq 0}2^{-N(j+k)-(d-1)|m|}\int_{|x|\sim
2^{m-n}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx\lesssim I_{1},$

we have that

$I_{2}\lesssim I_{1}+\sum_{n,j\geq 0}2^{-Nj+n(d-1)}\int_{|x|\lesssim
2^{-n}}|\mathcal{P}_{j+n}f(x)|^{p}dx$

holds. As before, we use the local orthogonality, Lemma 2.1 with parameter
$j$. Then, we obtain

$\sum_{j\geq 0}\int_{|x|\lesssim
2^{-n}}|\mathcal{P}_{j+n}f(x)|^{p}dx\lesssim\int|\mathcal{P}_{>n}f(x)|^{p}(1+2^{n}|x|)^{-N}dx$

for any $N>0$. By (3.14), we finally obtain

$I_{2}\lesssim\int|f(x)|^{p}|x|^{1-d}dx$

as desired, by (3.12).

Now we consider the other case, $k\geq n$. In this case, we have

(3.15) $\displaystyle\sum_{n\geq 0}\sum_{k\geq n}2^{-n}\int_{|x|\sim
2^{-k+n}}|\sum_{j\geq n}M_{j-n}f_{n}(x)|^{p}|x|^{1-d}dx$
$\displaystyle=\sum_{n\geq 0}2^{-n}\sum_{k\geq 0}\int_{|x|\sim
2^{-k}}|\sum_{j\geq 0}M_{j}f_{n}(x)|^{p}|x|^{1-d}dx.$

By Hölder’s inequality,

(3.16) $\displaystyle|\sum_{j\geq 0}M_{j}f_{n}(x)|^{p}$
$\displaystyle\leq(\sum_{j\geq
0}|M_{j}f_{n}(x)|^{p}2^{\delta_{0}|j-k|})(\sum_{j\geq
0}2^{-\delta_{0}|j-k|/(p-1)})^{p-1}$ $\displaystyle\lesssim\sum_{j\geq
0}|M_{j}f_{n}(x)|^{p}2^{\delta_{0}|j-k|}$

holds for $\delta_{0}>0$. Putting (3.16) into (3.15), we obtain

$\eqref{k>n eq1}\lesssim\sum_{n\geq 0}2^{-n}\sum_{k,j\geq
0}2^{\delta_{0}|j-k|}\int_{|x|\sim 2^{-k}}|M_{j}f_{n}(x)|^{p}|x|^{1-d}dx.$

By Proposition 1.2, we obtain

(3.17) $\displaystyle\int_{|x|\sim 2^{-k}}|M_{j}f_{n}(x)|^{p}|x|^{1-d}dx$
$\displaystyle\lesssim 2^{-\delta|j-k|}\int_{|x|\sim
1}|\mathcal{P}_{j}f_{n}(x)|^{p}|x|^{1-d}dx+\sum_{m\in{\mathbb{Z}}}2^{-Nj-(d-1)|m|-\delta
k}\int_{|x|\sim 2^{m}}|\mathcal{P}_{j}f_{n}(x)|^{p}|x|^{1-d}dx$
$\displaystyle\eqqcolon I_{3}+I_{4}$

when $p=2$ for $d\geq 3$, and when $p>2$ for $d=2$. We first see the
contribution from $I_{3}$. We choose $\delta_{0}=\delta/2$ so that we get
exponential decay. From the relation
$[\mathcal{P}_{j+n}f]_{n}=\mathcal{P}_{j}f_{n}$, we have

$\displaystyle\sum_{n\geq 0}2^{-n}\sum_{k,j\geq 0}2^{\delta_{0}|j-k|}I_{3}$
$\displaystyle\lesssim\sum_{n,k,j\geq 0}2^{-\delta|j-k|/2}\int_{|x|\sim
2^{-n}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx$
$\displaystyle\lesssim\sum_{n,j\geq 0}\int_{|x|\sim
2^{-n}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx.$

For each fixed $n\geq 0$, we use the local orthogonality. As above, by
covering $\\{x:|x|\sim 2^{-n}\\}$ by cubes with sidelength $\sim 2^{-n}$ which
are finitely overlapping and contained in a bigger annulus $\\{x:|x|\sim
2^{-n}\\}$, we have

$\displaystyle\sum_{j\geq 0}\int_{|x|\sim
2^{-n}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx$ $\displaystyle\lesssim
2^{n(d-1)}\int|\mathcal{P}_{>n}f(x)|^{p}(1+2^{n}|x|)^{-N}dx$
$\displaystyle\lesssim 2^{n(d-1)}\int|f(x)|^{p}(1+2^{n}|x|)^{-N}dx.$

By (3.14), the contribution from $I_{3}$ is as desired.

Now we see the contribution from $I_{4}$. Choosing sufficiently large $N$,
$\delta_{0}=\delta/2$, and the relation
$[\mathcal{P}_{j+n}f]_{n}=\mathcal{P}_{j}f_{n}$, we get

$\displaystyle\sum_{n\geq 0}2^{-n}\sum_{k,j\geq 0}2^{\delta_{0}|j-k|}I_{4}$
$\displaystyle\lesssim\sum_{n,k,j\geq
0,\,m\in{\mathbb{Z}}}2^{-Nj-(d-1)|m|-\delta k/2}\int_{|x|\sim
2^{m-n}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx.$

Splitting the summation in $m\in{\mathbb{Z}}$ by $m\geq 0$ and $m<0$, this is
bounded by the sum of the following two quantities:

$I_{5}=\sum_{n,j\geq 0}2^{-Nj+n(d-1)}\int_{|x|\lesssim
2^{-n}}|\mathcal{P}_{j+n}f(x)|^{p}dx,$ $I_{6}=\sum_{n,j,m\geq
0}2^{-Nj-m(d-1)}\int_{|x|\sim 2^{m-n}}|\mathcal{P}_{j+n}f(x)|^{p}|x|^{1-d}dx.$

Indeed, these are already bounded by $I_{2}$ and thus, we obtain

$\sum_{n\geq 0}2^{-n}\sum_{k,j\geq
0}2^{\delta_{0}|j-k|}I_{4}\lesssim\int|f(x)|^{p}|x|^{1-d}dx$

and this concludes the proof. ∎

##  4\. Proof of Proposition 1.2

The proof of Proposition 1.2 is different depending on whether $j\leq k$ or
$j>k$. In the former case, we use Lemma 2.6, the property of spherical
harmonics. In the latter case, we use the argument in [5].

###### Proof of Proposition 1.2 when $j\leq k$.

We first claim the following inequality for some $\delta(d,p)>0$ when
$d/(d-1)<p<\infty$:

(4.1) $\|\chi_{-k}(|\cdot|)M_{j}f\|_{L^{p}(|x|^{1-d})}\lesssim
2^{-\delta(k-j)}\|\mathcal{P}_{j}f\|_{L^{p}}.$

Note that the estimate does not depend on whether $k$ is bigger than $j$ or
not, and we have the $L^{p}$-norm without the weight on the right hand side.
We prove (4.1) with $\delta=0$ when $p=d/(d-1)$, and $\delta(d,2)>0$ when
$p=2$. Also, we have a trivial estimate with $\delta=0$ when $p\infty$. Then,
(4.1) follows by interpolation. When $p=d/(d-1)$, the estimate is already
proved in [5]. Indeed, inequality (3) in the proof of Theorem 10 in [5] has
$2^{j\epsilon}$-loss. However, authors provided sharp estimates without
$\epsilon$-loss restricting $x$ in a small annulus with width $2^{-k}$. Thus,
we only focus on $p=2$. Let

(4.2)
$\widehat{f}(\xi)=\sum_{k=0}^{\infty}\sum_{i=1}^{d_{k}}b_{i}^{k}(\rho)Y_{i}^{k}(\omega)$

for $\xi=\rho\omega$. By (2.1) and Lemma 2.2, it suffices to prove the
following:

(4.3) $\|\chi_{-k}(|x|)\int
e^{i(x\cdot\xi+t|\xi|)}\chi_{j}(|\xi|)\widehat{f}(\xi)d\xi\|_{L^{2}_{x,t}(|x|^{1-d};{\mathbb{R}}^{d}\times[1,2])}\lesssim
2^{\frac{d-2}{2}j-\delta(k-j)}\|\mathcal{P}_{j}f\|_{L^{2}}.$

Putting (4.2) into the square of the left hand side of (4.3), we get

$\displaystyle\int\Big{|}\chi_{-k}(r)\int
e^{i(r\rho\theta\cdot\omega+t\rho)}\chi_{j}(\rho)\sum_{k,i}b_{i}^{k}(\rho)Y_{i}^{k}(\omega)\rho^{d-1}d\rho
d\omega\Big{|}^{2}drd\theta dt$ $\displaystyle=$
$\displaystyle\sum_{k,i}\int\Big{|}\chi_{-k}(r)\int
e^{it\rho}\chi_{j}(\rho)b_{i}^{k}(\rho)\phi_{k}(-r\rho)\rho^{d-1}d\rho\Big{|}^{2}drdt$

by (2.5) and the orthogonality. For each $k,i$, Plancherel’s theorem implies

$\displaystyle\int\Big{|}\chi_{-k}(r)\int
e^{it\rho}\chi_{j}(\rho)b_{i}^{k}(\rho)\phi_{k}(-r\rho)\rho^{d-1}d\rho\Big{|}^{2}drdt$
$\displaystyle\leq$
$\displaystyle\int\Big{|}\chi_{-k}(r)\chi_{j}(\rho)b_{i}^{k}(\rho)\phi_{k}(-r\rho)\rho^{d-1}\Big{|}^{2}drd\rho$
$\displaystyle\approx$
$\displaystyle\int|\chi_{j}(\rho)b_{i}^{k}(\rho)|^{2}\rho^{d-1}\Big{\\{}\int|\phi_{k}(-r)\chi(2^{k-j}r)|^{2}\rho^{d-2}dr\Big{\\}}d\rho.$

By Lemma 2.6, we obtain

$\int|\phi_{k}(-r)\chi(2^{k-j}r)|^{2}\rho^{d-2}dr\lesssim 2^{(d-2)j+j-k}$

when $\rho\sim 2^{j}$, and the implicit constants are independent of $k,i$.
Thus, we can deduce (4.3) when $p=2$, and thus, completes the proof of (4.1).

Now we need to use localization arguments as in the proof of Lemma 3.1 to
reach the goal. Since $M_{c}$ is a local maximal operator, we obtain

(4.4) $\int_{|x|\sim
2^{-k}}|M_{c}[(1-\tilde{\chi})\mathcal{P}_{j}f](x)|^{p}|x|^{1-d}dx=0.$

As before, by the triangle inequality,

(4.5) $\displaystyle\int_{|x|\sim 2^{-k}}|M_{j}f(x)|^{p}|x|^{1-d}dx$
$\displaystyle\lesssim(\sum_{i=1,2}\|\chi_{-k}M_{c}[\mathcal{Q}_{j}^{i}\tilde{\chi}\cdot\mathcal{P}_{j}f]\|_{L^{p}(|x|^{1-d})}+\sum_{j^{\prime}>j+10}\|\chi_{-k}M_{c}[\mathcal{P}_{j^{\prime}}\tilde{\chi}\cdot\mathcal{P}_{j}f]\|_{L^{p}(|x|^{1-d})})^{p}.$

Note that $[\mathcal{Q}_{j}^{i}\tilde{\chi}\cdot\mathcal{P}_{j}f]^{\wedge}$ is
supported on $\\{\xi:|\xi|\sim 2^{j}\\}$, and
$[\mathcal{P}_{j^{\prime}}\tilde{\chi}\cdot\mathcal{P}_{j}f]^{\wedge}$ is
supported on $\\{\xi:|\xi|\sim 2^{j^{\prime}}\\}$ when $j^{\prime}>j+10$.
Thus, we can apply (4.1). As we have seen already, the main term will come
from $i=1$, and the other terms will be error terms. In the proof of Lemma
3.1, we obtained estimates for $\mathcal{Q}_{j}^{i}\tilde{\chi}_{k}$ and
$\mathcal{P}_{j^{\prime}}\tilde{\chi}_{k}$, (3.7) and (3.9). We use these
estimates for $k=0$. The calculation is almost the same with that of the proof
of Lemma 3.1. Thus, the proposition is concluded if we can handle the term
with $i=2$.

When $i=2$, we have

$\displaystyle\|\chi_{-k}M_{c}[\mathcal{Q}_{j}^{2}\tilde{\chi}\cdot\mathcal{P}_{j}f]\|_{L^{p}(|x|^{1-d})}$
$\displaystyle\leq\|\chi_{-k}M_{c}\mathcal{P}_{<0}[\mathcal{Q}_{j}^{2}\tilde{\chi}\cdot\mathcal{P}_{j}f]\|_{L^{p}(|x|^{1-d})}+\sum_{0\leq
m\leq
j}\|\chi_{-k}M_{c}\mathcal{P}_{m}[\mathcal{Q}_{j}^{2}\tilde{\chi}\cdot\mathcal{P}_{j}f]\|_{L^{p}(|x|^{1-d})}.$

We momentarily denote
$g(x)=\mathcal{Q}_{j}^{2}\tilde{\chi}\cdot\mathcal{P}_{j}f(x)$. The first term
on the right hand side is bounded by

$2^{\frac{k(d-1)}{p}}\Big{(}\int_{|x|\sim
2^{-k}}\Big{|}\int\frac{|g(x-z)|}{(1+|z|)^{N}}dz\Big{|}^{p}dx\Big{)}^{\frac{1}{p}}\lesssim
2^{-\frac{k}{p}}\Big{(}\int\frac{|g(x)|^{p}}{(1+|x|)^{N/2}}dx\Big{)}^{\frac{1}{p}}$

for any $N>0$. Also, the second term is bounded by

$\sum_{0\leq m<j}2^{-\delta(k-m)}\|g\|_{L^{p}}\lesssim
2^{-\delta(k-j)}\|g\|_{L^{p}}.$

Therefore, by the estimate (3.8), the proof is concluded. ∎

###### Proof of Proposition 1.2 when $j>k$.

Indeed, it is essentially proved in [5]. But we provide an alternative proof.
We first claim the following inequality for some $\delta(d,p)>0$ when
$d/(d-1)<p<\infty$:

(4.6) $\|\chi_{-k}(|\cdot|)M_{j}f\|_{L^{p}(|x|^{1-d})}\lesssim
2^{-\delta(j-k)}\|\mathcal{P}_{j}f\|_{L^{p}}.$

Again, (4.6) with $\delta=0$ when $p=d/(d-1),\infty$ is already known. Thus,
we choose a suitable $d/(d-1)<p<\infty$ and prove (4.6) with $\delta(d,p)>0$.
By (2.1) and Lemma 2.2, we may replace the left hand side of (4.6) by

(4.7)
$2^{j(-\frac{(d-1)}{2}+\frac{1}{p})+\frac{(d-1)k}{p}}\|\chi_{-k}(|x|)\mathcal{W}\mathcal{P}_{j}f\|_{L^{p}_{x,t}({\mathbb{R}}^{d}\times[1,2])}.$

We use the critical local smoothing estimate Proposition 2.5 when $d\geq 4$.
Precisely, we have

$\|\mathcal{W}\mathcal{P}_{j}f\|_{L^{p}_{x,t}({\mathbb{R}}^{d}\times[1,2])}\lesssim
2^{j(\frac{d-1}{2}-\frac{d}{p})}\|\mathcal{P}_{j}f\|_{L^{p}}$

when $p>2(d-1)/(d-3)$. This implies (4.7) is bounded by
$2^{-(d-1)(j-k)/p}\|\mathcal{P}_{j}f\|_{L^{p}}$. Therefore, (4.6) for the case
$d\geq 4$ is proved.

When $d=2,3$, it is not known that the critical local smoothing estimate holds
for $p\neq 2,\infty$. Instead, we use the critical $L^{p}$-$L^{q}$ local
smoothing estimate, Proposition 2.4.

Indeed, we have

$\|\mathcal{W}\mathcal{P}_{j}f\|_{L^{q}_{x,t}({\mathbb{R}}^{d}\times[1,2])}\lesssim
2^{j(\frac{d-1}{2}+\frac{1}{p}-\frac{d+1}{q})}\|\mathcal{P}_{j}f\|_{L^{p}}$

when $\frac{1}{p}+\frac{d+1}{(d-1)q}\leq 1$, $q>14/3$ for $d=2$ and $q>4$ for
$d=3$. This implies

$\|\chi_{-k}(|\cdot|)M_{j}f\|_{L^{q}}\lesssim
2^{j(\frac{1}{p}-\frac{d}{q})}\|\mathcal{P}_{j}f\|_{L^{p}}.$

By Hölder’s inequality, we have

$\displaystyle\|\chi_{-k}(|\cdot|)M_{j}f\|_{L^{p}(|x|^{1-d})}$
$\displaystyle\lesssim\|\chi_{-k}(|\cdot|)M_{j}f\|_{L^{q}_{x,t}(|x|^{-d+\frac{q}{p}})}$
$\displaystyle\lesssim
2^{-(j-k)(\frac{d}{q}-\frac{1}{p})}\|\mathcal{P}_{j}f\|_{L^{p}}.$

Therefore, we get (4.6) when $d/q>1/p$. By Proposition 2.4, we can choose such
$p,q$.

Now we use the localization argument to prove (1.1). We recall (4.4) and
(4.5). As before, we use (4.6) to
$\mathcal{Q}_{j}^{1}\tilde{\chi}\cdot\mathcal{P}_{j}f$ and
$\mathcal{P}_{j^{\prime}}\tilde{\chi}\cdot\mathcal{P}_{j}f$ when
$j^{\prime}>j+10$. Using (3.7) and (3.9), the proof is completely the same.
When $i=2$, the proof is simpler than the case of $j\geq k$. We just use the
boundedess of $M_{c}$ and obtain

$\|\chi_{-k}M_{c}[\mathcal{Q}_{j}^{2}\tilde{\chi}\cdot\mathcal{P}_{j}f]\|_{L^{p}(|x|^{1-d})}\lesssim
2^{\frac{k(d-1)}{p}}\|\mathcal{Q}_{j}^{2}\tilde{\chi}\cdot\mathcal{P}_{j}f\|_{L^{p}(1;B_{100})}$

where $B_{100}$ is the ball centered at the origin and radius $100$. The proof
is concluded by (3.8) since $j>k$ by choosing a sufficiently large $N>0$. ∎

## References

  * [1] D. Beltran, O Saari, _$L^{p}$ –$L^{q}$ local smoothing estimates for the wave equation via $k$-broad Fourier restriction_, J. Fourier Anal. Appl. 28 (2022), no. 5, Paper No. 76, 29 pp.
  * [2] J. Bourgain, _Averages in the plane over convex curves and maximal operators_ , J. Anal. Math. 47 (1986), 69–85.
  * [3] M. Cowling, J. García-Cuerva, H. Gunawan, _Weighted estimates for fractional maximal functions related to spherical means_ , Bull. Austral. Math. Soc. 66 (2002), no. 1, 75–90.
  * [4] J. Duoandikoetxea, M. Adela, O. Oruetxebarria, _The spherical maximal operator on radial functions_ , J. Math. Anal. Appl. 387 (2012), no.2, 655–666.
  * [5] J. Duoandikoetxea, L. Vega, _Spherical means and weighted inequalities_ , J. London Math. Soc. (2) 53 (1996), no.2, 343–353.
  * [6] L. Guth, H. Wang, R. Zhang, _A sharp square function estimate for the cone in $\mathbb{R}^{3}$_, Ann. of Math. 192 (2020), 551–581.
  * [7] Y. Heo, F. Nazarov, and A. Seeger, _Radial Fourier multipliers in high dimensions_ , Acta Math. 206, (2011), no. 1, 55–92.
  * [8] M. Lacey, _Sparse bounds for spherical maximal functions_ , J. Anal. Math. 139 (2019), no. 2, 613–635.
  * [9] J. Lee, S. Lee, _$L^{p}$ maximal bound and Sobolev regularity of two-parameter averages over tori_, arXiv:2210.13377.
  * [10] S. Lee, _Endpoint estimates for the circular maximal function_ , Proc. Amer. Math. Soc. 131 (2003), 1433–1442.
  * [11] G. Mockenhaupt, A. Seeger, C. Sogge, _Wave front sets, local smoothing and Bourgain’s circular maximal theorem_ , Ann. of Math. (2) 136 (1992), no. 1, 207–218.
  * [12] A. Nowak, L. Roncal, T. Szarek, _Endpoint estimates and optimality for the generalized spherical maximal operator on radial functions_ , Commun. Pure Appl. Anal. 22 (2023), no. 7, 2233–2277.
  * [13] L. Roncal, S. Shrivastava, K. Shuin, _Bilinear spherical maximal functions of product type_ , J. Fourier Anal. Appl. 27 (2021), no. 4, Paper No. 73, 42 pp.
  * [14] W. Schalg, _$L^{p}$ –$L^{q}$ estimates for the circular maximal function_, Ph.D. Thesis, California Institute of Technology (1996).
  * [15] W. Schlag, C. Sogge, _Local smoothing estimates related to the circular maximal theorem_ , Math. Res. Lett. 4 (1997), no. 1, 1–15.
  * [16] E. Stein, _Maximal functions: spherical means_ , Proc. Nat. Acad. Sci. USA 73 (1976), 2174–2175.
  * [17] by same author, _Harmonic Analysis: Real Variable Methods, Orthogonality and Oscillatory Integrals_ , Princeton Univ. Press, Princeton, NJ, 1993.
  * [18] E. Stein, G. Weiss, _Introduction to Fourier Analysis on Euclidean Spaces_ , Princeton Univ. Press, Princeton, NJ, 1975.
  * [19] by same author, _A sharp bilinear cone restriction estimate_ , Ann. of Math. (2) 153 no. 3, (2001), 661–698.