mmWave imaging system emits artificially generated signals to illuminate the target object, and 2D/3D images can be reconstructed by the reflected signals from the target according to the reflectance distribution. Different from the existing vision, IR, and X-ray imaging systems, mmWave signals can penetrate clothes and work in low visibility conditions. Due to its higher privacy and millimeter-scale ranging resolution, mmWave-based imaging has been widely pursued for pose/posture tracking<cit.><cit.>, automatic driving<cit.>, security concerns<cit.><cit.><cit.>, etc. We summarize the related works in TABLE <ref>.

The biggest challenge of mmWave imaging system is the data sparsity. The small size of the antenna array of mmWave radar device and its specularity lead to a  poor spatial resolution. In general, the imaging resolution of a radar system is defined by $resolution \propto wavelength \times distance/aperture$. For an antenna array size of 1.8 cm $\times$ 1.8 cm, its imaging resolution can only reach up to 28 cm at 1 m distance. To enhance resolution, a universal technique is the Synthetic Aperture Radar (SAR)<cit.>, in which the mmWave device needs to move a pre-determined trajectory and collect reflections from uniformly and densely grid locations. Recent works further propose to use the deep learning algorithm to generate images from mmWave signals.

The flowchart of mmFace.

mmFace<cit.> proposes a anti-spoofing face authentication (FA) system using a COTS mmWave radar (TI IWR1642-BOOST <cit.>). As Fig. <ref> illustrates, mmFace emulates a large aperture planar antenna array by moving a radar along a specific 2D slide rail, and collects the reflected signals from the human face. Firstly, as the reflection coefficient only depends on the material of the facial surface <cit.>, mmFace designs a novel algorithm to extract biometric features using the amplitudes of the reflected signals for liveness detection. With the reflected signals, a Range Migration algorithm (RMA) <cit.> is applied to construct a 3D facial image. However, it is distance-dependent and requires the authentication distance and registration distance to be identical, which is impractical. To further obtain distance-resistant features, mmFace utilizes the contour of the bright area on the facial image, which corresponds to the user's facial region and its surface curvature is relatively stable. This feature is utilized to calculate Fourier descriptors as the facial structure features for similarity-based user matching. Furthermore, to mitigate the registration overhead, mmFace also proposes a VRS (Virtual Registration Signals) generation method rather than directly collecting mmWave signals. Given three 2D facial photos taken from different perspectives, mmFace can reconstruct a virtual scene and generate the reflected mmWave signals by building a theoretical model of mmWave signal propagation. With the generated VRSes, the user's corresponding facial structure features are stored as templates. The experimental results that mmFace can achieve an average authentication success rate (ASR) of 96% and an average equal error rate (EER) lower than 5%.

Unfortunately, the pre-determined trajectory and uniform sampling make it extremely challenging to emulate SAR on a hand-held device. To this end, SquiggleMilli<cit.> uses a non-linear motion compensation and a compressed sensing-based framework to generate uniformly and densely spaced grid locations, as the traditional SAR imaging system does. As a user freely moves the device in the air, SquiggleMilli maps the measured sample to the nearest point on the uniform grid according to its phase change.  To avoid shape aliasing from the missing grid samples, SquiggleMilli estimates the missing by combining several measurements around its corresponding location. The recovery procedure then can be represented as an L1-norm minimization problem and solved by the unconstrained basis pursuit de-noising method <cit.>. Then it applies voxel segmentation to extract the 3D shape of objects at different depths. To enhance the perceptibility of the reconstructed 3D shape, SquiggleMilli  leverages a pre-trained cGAN<cit.> model to recover the high spatial frequencies in the object to generate an accurate 2D shape with the partial 3D mmWave shape. It further designs two networks to predict its 3D features and category. The experimental results show that SquiggleMilli can reconstruct 2D shapes with a similarity score ranging from 0.85 to 0.95. For the 3D feature, it can predict the mean depth and rotation angle with less than 1% error and 1.5$^{\circ}$ error for the 90-th percentile. Meanwhile, it achieves more than 90% accuracy for classifying the objects.

Different from SquiggleMilli, milliCam <cit.> proposes a mmWave-based scanning system for capturing a high-resolution shape of a small metallic object by emulating the SAR imaging principle and swiping the hand-held mmWave device over the air. As SAR requires stringent linear motion and mm-scale localization accuracy, milliCam first leverages the co-located camera to compute the position and tractory of the device by measuring the translation and rotation. The camera has sub-mm pixel resolution and can achieve a 7mm localization error. Considering the target-scene is very close-by, such localization errors still cause significant errors. milliCam observes that most of the target-scene in the mmWave signals are sparse, due to the specular reflection. It re-designs a novel motion compensation algorithm based on the airborne SAR compensation by fusing a clustering algorithm and a bicubic interpolation. Furthermore, to resist the varied reflectivity and noise reflections, milliCam iteratively segments and corrects the defocused image using Otsu's method and squint correction, separately. After that, in order to ensure the phase coherency across the emulated apertures, milliCam calibrates the incoherent phases relying on the line-of-sight signals from Tx to Rx. Results show that the measurement errors are less than 20% and 4% for $5 \times 5$ cm$^{2}$ and $20 \times 20$ cm$^{2}$, respectively.

The flowchart of milliCam.

However, autonomous driving cars move fast and their trajectories are uncertain, making it very challenging for precise localization at the scale of the signal wavelength and uniform sampling, which are the requirements of emulating the SAR technique. To this end, MILLIPOINT <cit.> enables SAR imaging on low-cost commodity radar to generate 3D point clouds. First, MILLIPOINT observes that different Tx/Rx pairs may experience similar channel responses with a lagging effect, and the delay depends on the spacing between antenna pairs. It characterizes the radar's cross-movement by correlating the received signals. The correlation reaches the maximum at the range of half a spacing of two antenna pairs. In addition, MILLIPOINT uses multiple antenna pairs with different spacing to overcome the influence of side peaks and achieves robust self-tracking by applying dynamic programming.

Furthermore, due to the existence of specular reflection, only reflections around the normal direction of the target can be received by the radar, resulting in a very small effective aperture. To synthesize a full image of the 3D environment, MILLIPOINT proposes an automatic multi-focusing mechanism to focus on each target separately and then synthesize them. To achieve it, MILLIPOINT first compensates for the antenna gain and the attenuation in the cross-range spectrums. It further identifies targets as the peaks of frequency spectrums along the range direction by an empirical threshold. The corresponding peak widths can be regarded as the target's effective aperture lengths. Then the classical SAR imaging algorithm is applied and synthesizes a 2D image by selecting pixels with the maximum value across all images. To extend it with height information, MILLIPOINT models the relationship between height and phase response of pixel, according to the idea of AOA estimation with antenna array <cit.>. The height is estimated with the Capon algorithm <cit.>. Through evaluation of the commercial mmWave Radar with 6 Tx and 8 Rx antennas, results show that MILLIPOINT achieves the median cumulative tracking error of 1.2% and has a precision rating of sub-mm.

As the SAR-based imaging methods require the target to be stationary or relatively stationary, they are not suitable for tracking the target's posture that dynamically changes. However, the spatial relationships and temporal dependencies of the joint skeletons are concealed in the sequence of mmWave reflected signals. This information can be utilized for pose tracking. MilliPose  <cit.> proposes a machine-learning model for pose tracking and solves the problems of poor resolution, specular reflection, and varied reflectivity. Firstly, MilliPose leverages cGAN to generate a high-resolution 2D full-body silhouette image from the low-resolution 3D mmWave signals. Meanwhile, Considering that the degrees of freedom (DoF) and the ranges of body joints are very limited, MilliPose designs an RNN, consisting of a two-layered GRU network <cit.> and a Structured Prediction Layer (SPL) <cit.>, to learn these rules and predict skeleton joint of the next pose. Then it feedbacks the predicted pose to cGAN and can generate a high-quality body shape. Results show that the median error of joint location prediction is only 2.1 cm.

Instead of applying SAR imaging, mmFER<cit.> develops a mmWave radar-based system to recognize facial expressions by extracting and exploring subtle facial muscle movements from raw mmWave signals. To this end, mmFER first leverages the MIMO technique to localize the facial areas of the target and eliminate ambient noise. Then it utilizes Gaussian Mixture Model (GMM) to mitigate the impact of body motions on facial localization. mmFER further proposes a cross-domain transfer pipeline to guarantee an effective and reliable model knowledge transformation from image to mmWave. In particular, a hybrid learning loss function is designed to effectively address the over-fitting issue caused by small-scale training datasets. Moreover, an autoencoder is explored to learn the transition of latent features to reshape mmWave data, thereby eliminating the influence of data heterogeneity. mmFER is fully implemented using a commercial mmWave radar (TI IWR1843BOOST), and results show that mmFER obtains an accuracy of at least $80.57\%$ when the distance between the subject and radar is within the range of 0.3-2.5 m.

Different from generating 2D silhouette images,

mmMesh<cit.> proposes a deep learning framework to reconstruct the dynamic 3D human mesh from the mmWave signals. To alleviate the influence of ambient noise and multipath effects, mmMesh uses an attention mechanism to discriminate the point cloud and the points reflected by the subject tend to have higher qualities. Then a rough estimation of the shape and pose can be generated. To correctly align the points and make the human mesh more accurate, mmMesh dynamically chooses some "virtual locations" near the subject as anchor points that are each related to a part of the human body. According to the anchor locations, mmMesh dynamically groups the 3D point cloud into several subsets, and each one corresponds to a different body segment. Meanwhile, the local structure and associations can be learned. Finally, mmMesh incorporates the Skinned Multi-Person Linear (SMPL)<cit.> model as an additional constraint. It allows us to represent the whole 3D human mesh with 86 parameters and does not need to calculate the positions of thousands of vertices. The experimental results show that mmMesh can achieve an average vertex error of 2.47 cm and a mesh localization error of 1.27 cm. Besides, the average joint localization and rotation errors are 2.18 cm and 3.8 degrees, respectively.

Besides, RPM <cit.> designs a multi-dimensional feature fusion backbone network for human pose estimation. It consists of two blocks: one is a channel fusion block that can effectively fuse horizontal and vertical RF features based on the channels' correlation. The other one is a deformable multi-stage convolution module that maintains scale-insensitive feature representations. Due to the high directivity of mmWave radar, it is possible to receive the reflected signal of a subset of limbs at each RF snapshot. Therefore, a spatial attention module is applied to recover the remaining body parts from a single RF snapshot by modeling non-local skeleton relationships. A temporal attention module is further applied to model the temporal dependencies across the 3D skeleton sequences. Finally, the complete 3D skeleton can be obtained. The experimental results reveal that RPM can achieve a Mean Per Joint Position Error (MPJPE) of 5.71 cm.

The architecture of HawkEye.

Although deep learning-based methods are proven to be effective, the procedure of collecting real-world signals for training is very time-consuming and labor-intensive. Benefiting from the success of the GAN technique and its robust resistance to noise, HawkEye <cit.> uses the GAN technique to generate the mmWave signals for training. In this work, a cGAN-based architecture is designed to predict high-resolution depth maps with a low-resolution mmWave heatmap input. HawkEye utilizes an encode-decode framework as the generator network to generate a 2D depth image. Meanwhile, the skip connection mechanism <cit.> is applied to retain the high-frequency details in depth. The discriminator employs a two-stream architecture to classify real vs. generated samples. Furthermore, to avoid collecting real-world mmWave data, HawkEye proposes a data synthesizer to simulate the propagation of the mmWave signals in the real world using ray-tracing <cit.> and generate realistic mmWave heatmaps.

Different from the use of radar devices, mmEye<cit.> is the pioneering work of attempting to achieve mmWave imaging on a single commodity 60GHz WiFi device. To break through the resolution limited by the physical aperture, mmEye proposes to reconstruct the spatial spectrum by applying the Multiple Signal Classification (MUSIC)<cit.> with Akaike information criterion (AIC)<cit.>, a well-known information-theoretic approach for model order selection. However, The rank of MUSIC is likely much less than the number of incoming signals, making its performance greatly deteriorate or even fail. To overcome the rank deficiency problem without loss in precision, mmEye designs a novel 2D spatial smoothing termed Joint transmitter smoothing (JIT). This technique reuses all the Rxs for each Tx rather than dividing the receive array. mmEye further proposes a background and noise cancellation (BANC) algorithm to remove interfering signals and a minimum mean square error (MMSE)<cit.> estimator is applied to compute the background CIR. Additionally, mmEye searches the point of interest (PoI) based on the variance of the energy distribution of the spatial spectrum. Finally, mmEye transforms the PoIs into plain images by solving a weighted least absolute deviation problem. The experimental results show that it achieves a median silhouette (shape) difference of 27.2% and a median boundary keypoint precision of 7.6 cm.

Although commodity RF signals have been utilized to realize 3D human posture tracking, these studies cannot handle cases where multiple users are in the same space. Towards this end, $m^3$Track<cit.> proposes a 3D posture tracking system. Based on the varying  Doppler responses along with the speed of movement, $m^3$Track performs a convolution operation on the Range-Doppler-Profile to detect all users with their corresponding ranges. In the meanwhile, it utilizes minimum variance distortionless response (MVDR)<cit.> to separate all users from different angles. Given the range and the angle, $m^3$Track extracts the mmWave profiles of each user based on a 3-cylinder model, relating to the head-neck, chest-arm and lap-leg, respectively. Each profile contains two Range-Angle-Profiles (spatial features) and one Range-Doppler-Profile (temporal features). Then, $m^3$Track designs a two-steam deep learning architecture. It takes the spatial features and temporal features as inputs to extract the global body shape features and local body motion features. By concatenating these features, a regression model is applied to predict the 3D coordinates of skeleton joints and reconstruct the 3D body posture. For tracking posture in multi-user scenarios, it is necessary to map the reconstructed posture into real-world 3D space accurately. $m^3$Track leverages the K-means algorithm <cit.> to separate the point clouds into multiple clusters. Each cluster center corresponds to a posture joint, and the optimal mapping relation can be obtained by searching the minimal mapping error. Considering that the above computed 3D coordinate is in the polar coordinate system, $m^3$Track further designs a coordinate-correlated extended Kalman filter to calculate the user's position in the Cartesian coordinate system. With the position trajectories, $m^3$Track realizes continuous 3D posture tracking in real-world multi-user scenarios. With TI AWR1443 <cit.>, the experimental results show that the overall joint tracking error is 42.4 mm in a 4-user scenario.

§.§ Lessons Learned

The current mmWave-based human sensing has covered a variety of sensing tasks and corresponding solutions. When people try to utilize mmWave sensing to achieve a certain sensing task, they should first choose the appropriate hardware platform and processing signal form. For example, the point cloud is more suitable for contour-related sensing tasks such as activity recognition or human imaging, while the phase waveform is more suitable for fine-grained quantitative sensing tasks such as handwriting tracking or biometric measurement.

As we mentioned before, different sensing tasks have different technical frameworks and thus different challenges. For example, tracking and localization works are still plagued by trajectory crossing and multipath effect, which leave significant room for improvement in indoor multi-person localization. In terms of motion recognition, continuous motion recognition and dynamic human orientation changes are still open problems. Considering the unique requirements of biometric measurements for localization and sensing of human body parts (chest, throat, etc.), precise body parts localization and enhancement of the weak biometric signals are worthy of research, due to the uncertainty of human body position and posture. In human imaging, since SAR technique and deep learning models are widely used, how to improve the human imaging speed and model generalization is worthy of further study.

People then are encouraged to explore the core technical challenges of the corresponding sensing task, which determine the detailed technical targets of the algorithms to be adopted. For example, in recognition tasks such as activity recognition, gesture recognition, and voice recognition, considering the data differences (duration, intensity, variation, etc.) brought by different individuals, the learning-based classification methods often show better recognition abilities. Moreover, in measurement tasks such as handwriting tracking and vital sensing, the complex noise has a significant impact on sensing results, due to the requirement for fine-grained quantitative measurements. Therefore, appropriate noise reduction methods need to be carefully designed.

Furthermore, the sensing scenes should be carefully observed to assist sensing methods and achieve better sensing results. For example, human orientation and radar deployment need to be carefully considered to reduce potential signal distortion and environmental noise. On the other hand, multiple reflections between human and environmental reflectors and the impact of the human body on the environment may contain additional human-related information, which can help to achieve more general or accurate sensing.

Finally, people are encouraged to deploy their systems in various scenarios to verify their performance in detail. Considering the increasing demand for fine-grained sensing in complex scenarios, such as in smart cockpits and smart homes, multi-scenario experimental verification can not only explore the potential capabilities but also promote the deployment of mmWave-based human sensing in actual scenarios.

§ CHALLENGES & FUTURE DIRECTIONS

According to the discussion in the previous section, we can see that there is still a big gap between the state of the arts and the ideal vision of mmWave-based human sensing, with respect to the capacity, applicability, accuracy, and ubiquity. Filling this gap requires continuous innovations in all the dimensions around this technology, such as hardware, algorithm, sensing mediums, and application designs. Therefore, our discussion in this section will accordingly focus on four main research directions, namely hardware and platforms, enhancing the applicability, novel sensing schemes, and integration with new mediums.

§.§ Hardware and Platforms

§.§.§ Current limitations

The current mmWave-based sensing technology still deeply suffers from limited hardware capabilities. The insufficient number of antennas, limited transmission power and hardware denoise capability hinder the further development of mmWave-based sensing.

Taking mmWave radar based sensing as an example, the COTS mmWave radar only has several antennas (e.g., 3 TX antennas and 4 RX antennas in IWR1443 and IWR6843ISK-ODS), which leads to very limited angle resolution (15$^\circ$). The reason is that these radars can only provide $3*4=12$ virtual antennas by letting 3 Tx antennas take turns transmitting signals and 4 Rx antennas receive signals simultaneously. Then the Angle-FFT operation can obtain an angle resolution of $\frac{180^\circ}{3*4}=15^\circ$ <cit.>. Such angle resolution is insufficient to enable many fine-grained sensing tasks, such as face imaging. Researchers have attempted to perform Rx beamforming <cit.> and synthetic aperture radar (SAR) technique <cit.> to expand the radar's angle resolution. However, they either have limited improvements or need a customized slide rail, causing the current solutions still be unsatisfactory. To further enhance the angle resolution, it's necessary to increase the number of antennas and exploit novel signal enhancement techniques. Texas Instruments has launched a powerful radar, TI MMWCAS-RF-EVM <cit.>, which has 12 Tx antennas and 16 Rx antennas and has a much better angle resolution. However, its four-radar cascaded array implementation makes its form factor multiply. Moreover, its power consumption (higher than 20 W <cit.>) is much higher than that of those radars mentioned before (less than 2 W <cit.>). These factors limit the practical deployment of this radar, especially in mobile scenarios where energy consumption needs to be carefully considered. In fact, the impact of its high energy consumption has been discussed in the autonomous driving community. Moreover, some signal local resolution techniques, such as Chirp-Z transform techniques <cit.>, can be explored.

The sensing range is another important technical indicator. Although the mmWave radar can sense vehicle targets from tens of meters, the range of human sensing is very limited. For example, mmTrack <cit.> can only track users within 5 m range. WaveEar <cit.> localizes the position of the speaker and performs speech recognition within 2 m range. The reason is that the extremely high frequency of the mmWave causes its rapid attenuation, which makes the human-related signal features blurred a few meters away. To overcome the limitation of sensing range, more powerful equipment is needed. They can be radars with higher transmission power or some auxiliary equipment that can enhance the reflected signals, such as retro-reflective tags <cit.>.

To obtain more accurate sensing results, careful signal noise reduction is necessary to resist hardware deficiency and environmental interference. However, the existing hardware denoise capability cannot fully meet our needs. For example, mmSpy <cit.> models the noise from the ramp/settle times in the radar's oscillator. It further employs statistical error correction techniques to subtract it from the phase waveform. These imperfect hardware features may affect our sensing results and must be carefully modeled and handled.

§.§.§ Potential directions

Considering that the current limited hardware capabilities seriously constrain the development of sensing techniques, we hope to discuss the development trend of the future mmWave hardware to provide some ideas and reasonable imagination for researchers in industry and academia.

We think the future mmWave hardware will have the characteristics of miniaturization, an increased number of antennas and flexible customization, etc. With the continuous development of the radio-frequency integrated circuit (RFIC) technique, the RF front-end and computing units of the radar have been able to be integrated into a small-size chip. In this way, the mmWave hardware can be easily integrated into mobile phones, smartwatches and other devices to achieve more ubiquitous sensing. In the past few years, Soli <cit.> has made this attempt but most radars still have not been integrated. Nevertheless, we believe that miniaturization will be a major trend of mmWave hardware.

Furthermore, since the number of antennas determines the angle resolution and directional sensing capability of the mmWave hardware, more antennas will definitely be welcomed in industry and academia. Texas Instruments has noticed this extensive demand and launched a more powerful radar, TI MMWCAS-RF-EVM <cit.>, which forms a $12 \times 16$ virtual antenna array. Compared with mmWave radars, mmWave probes usually have more antennas, such as the 32 Tx antennas and 32 Rx antennas of Qualcomm 802.11ad 60GHz WiFi. However, these probes cannot easily form a larger virtual antenna array. In addition, the sensing pipeline of transmitting and receiving in turn also limits their sensing frequency.

Finally, we believe that more flexible customization is attractive and fascinating. The current mmWave hardware platform has achieved a lot of signal collection and processing functions. However, due to the high integration of hardware, it is difficult for developers to modify the hardware to make their attempts, such as antenna layout and polarization direction modification. We believe that more flexible customization can certainly stimulate the community creatively.

§.§ Enhancing the Applicability

§.§.§ Challenges in real deployments

In addition to limited hardware capabilities, complex deployment environments also bring various critical challenges to mmWave-based human sensing. NLoS scenarios and multipath effects are the most common challenge in these complex environments.

Due to the human's position and posture dynamics and the blockage between the radar and the target, the human-related signal sometimes cannot be obtained directly, especially in NLoS scenarios. In this case, we can obtain the human-related signal with environmental reflection <cit.> or realize indirect sensing through the influence of the human on surrounding objects <cit.>. Some works also try to utilize the penetrability of the mmWave signal to achieve human sensing in NLoS scenarios <cit.>. However, their application scenarios are limited due to insufficient penetrability.

Multipath effects also challenge the actual deployment of mmWave-based human sensing, especially in dense environments. Due to the reflection of mmWave signals on the surface of environmental objects, the received signal is the superposition of multiple signals with different time delays and attenuation. Researchers have attempted to eliminate multipath interference by either deploying multiple radars <cit.> or exploiting the consistency of the dynamic between the target and the ghost images <cit.>. Some works begin to utilize rather than eliminate the multipath effect. For example, mmReliable <cit.> utilizes multiple beams to achieve more reliable communication links. However, the exploitation of multipath effects in human sensing remains to be explored.

In addition, large-scale ground truth data is indispensable to the actual deployment of mmWave-based human sensing. Although the existing works have provided a lot of datasets, as mentioned in Sec. <ref>. Datasets that contain quantitative sensing results are lacking. For example, the actual measurement dataset of human respiration and heartbeat is significant to develop the relevant mmWave-based sensing techniques.

§.§.§ Promising directions

To resist the complex deployment environment and provide accurate sensing results, researchers have made many various efforts <cit.>. Based on those works, we discuss the promising directions that appear to be effective in enhancing the applicability of mmWave-based human sensing. Retracing the continuous enrichment of mmWave-based human sensing applications in the past few years, we think that the future mmWave sensing applications will be more quantitative, ubiquitous and fine-grained, etc.

Although mmWave-based human sensing has realized a variety of sensing applications, varying from tracking and localization to vital sign sensing, the quantitative sensing of the human body still needs to be developed <cit.>. Taking gesture recognition as an example, the existing works can well distinguish the gesture types. However, the quantitative indicators such as the amplitude of each gesture are difficult to analyze. To further promote the development of mmWave sensing techniques, accurate quantization results are needed. Moreover, quantitative sensing is a key step to realizing the practical application of sensing techniques. Without quantitative indicators that meet the error standard, it is difficult for the public to pay for these sensing techniques.

Another development trend of mmWave sensing applications lies in ubiquitous sensing, including wider sensing ranges, more universal sensing scenarios and richer sensing capabilities. As we mentioned before, due to the complex deployment environment, most of the existing works are the realization of single sensing tasks within the limited sensing range of specific scenarios. How to achieve more ubiquitous sensing is still an open problem. In the actual scenes, many factors such as range, occlusion, multipath and so on will lead to the limited application of mmWave sensing. Some works have tried to expand the sensing application scenarios <cit.>. However, more efforts are needed for this vision.

Finally, with the deepening of mmWave sensing techniques, more and more fine-grained sensing applications have been developed, including vital sensing and sound recognition, etc. However, we believe that the potential of mmWave sensing techniques goes beyond this. With directional and fine-grained sensing capabilities, mmWave sensing techniques are expected to enable more sophisticated sensing applications, such as skin disease detection, blink recognition, etc.

§.§ Novel Sensing Schemes

To further enrich the mmWave-based sensing capabilities, researchers have begun to find novel sensing schemes, such as fusion sensing and multitask sensing. Furthermore, researchers have noticed the indirect sensing ability of mmWave sensing and tried to realize enhanced sensing through “sensing side channels”.

§.§.§ Fusion sensing

After realizing the limitation of existing mmWave-based human sensing techniques, some researchers begin to explore the feasibility of fusion sensing. By combining mmWave signal with other sensing media, such as vision, inertial measurement unit (IMU) and acoustic, fusion sensing is expected to provide more accurate and robust performance.

On the one hand, the sensing efficiency and ability can be increased by fusion sensing. The sensing techniques based on other media can share a part of the sensing task for mmWave sensing. For example, <cit.> utilizes a mmWave + camera multimodal sensing system to track the box's position and the relative position of fragile products in the box. The vision-based sensing algorithm is employed to locate the box along the conveyor belt, making the mmWave sensing can focus on computing the product's location.

On the other hand, by mutual correction with the sensing results based on other media, fusion sensing can further improve the sensing accuracy. <cit.> employs a mmWave radar based simultaneous localization and mapping (SLAM) solution assisted by IMU. The IMU data is utilized to combine continuous radar scanning point clouds into “multi-scan” to achieve accurate and robust SLAM results. Wavoice <cit.> explores the inherent correlation between the audio sensing result from a mmWave radar and a microphone. By combining the noisy audio signals and the aimless mmWave signal, Wavoice can achieve noise-resistant speech recognition. Considering the complexity of human sensing, fusion sensing is an attractive and effective way.

§.§.§ Multitask sensing

The complexity of human sensing is also reflected in the interlacing of human sensing tasks. For example, both gesture recognition and vital sensing are important components of health monitoring in smart home. If we can sense these two tasks simultaneously with a single mmWave radar, the deployment and implementation of smart home can be simplified, and the constraints on people will be further released.

However, multitask sensing is not simply the superposition of multiple single-task sensing. For example, human gestures and vital signals are closely related and are difficult to be separated. The relation between multiple human sensing tasks is a challenge to achieve multitask sensing, which makes it difficult to describe and analyze the reflected signals. Conversely, it is also an opportunity as mutual correlation and improvement can be performed. In this case, the multitask learning techniques <cit.>, which have made great progress in the past ten years, may be able to show unique advantages in multitask sensing.

§.§.§ Side channel for human sensing

In addition to directly sensing the human body, there are many potential “sensing side channels” that can be used to obtain human-related information. By analyzing the influence of the human body on other objects, we can deduce the characteristics of the human body itself.

The current sensing side channel is most commonly used in sound recognition especially in eavesdropping. mmEve <cit.> and mmSpy <cit.> attempt to recover speech emitted from smartphone earpieces. They both utilizes the fact that the reflected mmWave signals from the smartphone's rear are highly correlated with the soundwaves emitted from the smartphone's earpiece. Similarly, mmPhone <cit.> utilizes the changes of piezoelectric films with sound pressure to decode the speech through soundproof obstacles. There are also some works to eavesdrop devices with “sensing side channels”. For example, SpiralSpy <cit.> utilizes a malicious software to encode data into the fan control signals. Then the fan motion status can be sensed and decoded by a mmWave radar.

Considering that the influence of the human body on the surrounding environment does not only exist in voice but also in many other physical signs, researchers can further explore the sensing side channel in many sensing tasks, such as keyboard sensing and gait recognition.

§.§ Integration with New Mediums

Innovations in the wireless medium is the extra boost of mmWave-based sensing.
Emerging wireless communication techniques are deemed to provide ideal sensing mediums. With the continuous development and innovation of wireless communication techniques, many of them have been applied to sensing tasks to achieve ubiquitous sensing, including integration sensing and communication, backscatter, intelligent reflecting surface and THz sensing.

§.§.§ Integration sensing and communication

With the increasing demand for human sensing and the number of mmWave network techniques such as 3GPP 5G-NR and IEEE 802.11ad/ay, Integration sensing and communication (ISAC) has received extensive attention from academia and industry.

Given the growing popularity of mmWave networks, human sensing based on communication devices seems to be more suitable for life scenes. Chenshu Wu et al. propose a series of works based on 60GHz probe, such as mmWrite <cit.>, ViMo <cit.> and mmTrack <cit.>. However, these works all need to adjust the probe to a radar-like mode to give up the communication function. Therefore, such a design can only support the rotation of sensing and communication but can not achieve ISAC. On the contrary, SPARCS <cit.> proposes an IEEE 802.11ay based ISAC solution. It utilizes the CIR measurements from sparse communication data packages to analyze human movement without interfering with communication. Considering the popularity of 5G network and WiZig, ISAC has great potential to be applied to daily scenarios.

Moreover, some researchers have begun to endow the mmWave radar, which is customized for sensing, with additional communication capabilities. For example, people have changed the frequency, transmission time and interval of the mmWave signal to encode information. We will not expand as this is beyond the scope of this survey.

there are several critical technical challenges in implementing ISAC, both communication-assisted sensing and sensing-assisted communication. Firstly, people need to carefully consider the trade-off between information delivery and channel state estimation. The latter usually serves as the basis for sensing. More power allocated to information delivery means less sensing capability, and vice versa. Since the ratio of communication and sensing is different in different scenarios, people should analyze the trade-off by actual measurement and modeling to achieve the desired target. Secondly, due to hardware limitations and environmental interference (e.g., LoS path blockage), the coverage of ISAC is often limited, which hinders the application of ISAC in many scenarios. To solve this problem, new auxiliary devices such as Intelligent Reflecting Surface (IRS) can be considered to be deployed around the main ISAC device to further expand the coverage of ISAC. For example, Mosiac <cit.> utilizes multiple curved reflectors to provide comprehensive coverage to achieve omnidirectional around-corner automotive radars. Finally, the interference from other devices will be stronger in ISAC compared to pure wireless sensing scenarios. The interfering signals can cause the ISAC signals to be corrupted, delayed or even lost. To improve the robustness of ISAC in scenes full of interference, people can either adopt compensatory operation similar to SPARCS <cit.> or protect ISAC signals from interference through scheduled measures, such as frequency division multiplexing and time division multiplexing.

§.§.§ Sensing over backscatter

Backscatter technology <cit.> has been developed rapidly in the past few years to provide communication capability for low-end devices. Compared with active radios, backscatter <cit.> promises to be extremely low power, smaller and cheaper alternative. As the reflected signal of the backscatter tag is affected by both the modulation information and the channel state, we can sense the target state by attaching the tag to the target and analyzing the change of the reflected signal.

Backscatter technology has been used to improve the sensing range of mmWave sensing. For example, Millimetro <cit.> exploits a customized backscatter tag to realize accurate localization in a long range. The tag is formed by Van Atta retro-reflectors, which can reflect the carrier signals back in the direction of arrival regardless of the incident angle and the tag's mobility. With the retroreflected tag, The attached target can be localized at high accuracy (centimeter-level) over extended distances (over 100 m). Such a design can enhance the mmWave reflected signal and expand the sensing range of mmWave sensing.

The mmWave sensing capability can also be enhanced by combining mmWave sensing and backscatter technology. RoS <cit.> utilizes multiple configurable Van Atta arrays to build radar readable road signs. The arrays are formed as customized geometrical layouts and can be distinguished by an automotive radar.
With the continuous development of mmWave backscatter technology <cit.>, we believe that there will be more mmWave sensing works combined with backscatter technology adapting to various sensing tasks.

§.§.§ Sensing with intelligent reflecting surface

As a cost-effective technology, Intelligent Reflecting surface (IRS) <cit.> has been an attractive subject to improve the performance of communication and sensing systems by reconfiguring the wireless propagation environments smartly. Generally, IRS is a metasurface consisting of a large number of passive reflecting elements. Each reflector induces an adjustable phase shift and amplitude variation to the incident signals. With the ability to reconfigure the direction of mmWave signals, IRS is expected to play an important role in enhancing the coverage and energy efficiency of mmWave systems.

Due to the strong directivity and rapid attenuation of mmWave signals, the coverage of mmWave sensing is very limited. As IRS can reconfigure the mmWave signal direction, it can guide the mmWave signal to the originally inaccessible positions to enhance the sensing coverage. For example, MilliMirror <cit.> designs a passive metasurface prototype which expands the coverage of mmWave radios to blind spots by redirecting and reshaping mmWave signals to any anomalous directions. mmWall <cit.> proposes another tunable smart surface made of metamaterial, which enables a fast mmWave beam relay through the wall and redirects the beam power to another direction when a human body blocks a line-of-sight path. In addition, mmWall also supports splitting the incoming signal into multiple beams and concurrently steering the multi-armed beams. In this way, “through-wall” sensing or even multi-person sensing can be performed.

With such powerful capability, however, IRS has the potential to be utilized for illegal attacks. MeSS <cit.> has achieved illegal eavesdropping based on mmWave metasurface. By leveraging two degrees of freedom (space and time) in reconfigurable surfaces, MeSS generates and steers a concealed directional sideband toward the eavesdropper while maintaining the direction of the mainband toward the legitimate client. Joonas Kokkoniemi et al. <cit.> also discuss the channel modeling of the phased array type reconfigurable IRSs in the mmWave band.

To sum up, we believe the research of IRS has the opportunity to open a novel direction for solving the pain points of mmWave sensing, such as NLoS, mobility, and so on. Meanwhile, it also brings new challenges to our privacy protection.

§.§.§ From mmWave to THz

With the rapid increase of smart applications and devices, it is foreseeable that 5G network is hard to meet the ever-increasing network traffic. According to ITU-T's estimation <cit.>, the global mobile traffic will grow to 5016 EB per month in 2030, compared to 62 EB per month in 2020. To solve this problem, the 6G network has been discussed and researched by industry and academia. mmWave signal and THz signal are expected to be used in 6G because of their wide unexplored spectrum. At present, mmWave communication and THz communication have reached 1 terabyte/s and 206.25 Gbit/s data rate in laboratory environments <cit.>.

On the other hand, due to the lack of corresponding standards, we make some predictions about the requirement of 6G in human sensing from application scenarios. The 6G network is expected to be human-centric and provide comprehensive human sensing. Therefore, 6G will pay more attention to various human-related applications, such as holographic-type communications, digital twins and tactile internet. Taking the tactile internet as an example, human sensing techniques may be used to recognize human body movements and the 6G network is expected to meet the low transmission delay of 1 millisecond or less reaction time <cit.>. Based on our summary of mmWave sensing, we give some discussions on THz sensing to complement blueprint for 6G.

THz signals share many similarities with mmWave signals, including high frequency, large bandwidth and extremely high attenuation. With the continuous development of wireless sensing, THz signals have also begun to enter the field of vision of researchers and are used to realize some sensing tasks, such as indoor positioning <cit.>. Since THz signal frequency (0.3-3 THz) is much higher than that of mmWave signal (0.03-0.3 THz), THz sensing faces many unique challenges and has attractive potential capabilities.

The high-frequency characteristics of THz signals will cause extremely higher attenuation, resulting in very limited sensing coverage and low-SNR received signal. The limited sensing coverage may make it difficult to apply THz sensing to outdoor human sensing. Furthermore, how to reliably extract the human-related phases from the low-SNR received signal needs to be further explored. Novel low-noise electronics may be demanded. In addition, considering the separated transceiver device, tight synchronization is needed to extract human-related information, which will be more challenging in higher frequencies.

Conversely, the higher frequency and larger bandwidth of THz signals enable THz sensing to have finer sensing capability and higher spatial resolution, which means that more fine-grained sensing tasks can be achieved, such as fine-grained medical imaging, fingerprint detection, skin texture detection, etc. Considering the frequency limitation of mmWave signals, THz sensing is more suitable for achieving such fine-grained sensing tasks.

§ CONCLUSION

mmWave sensing has become more and more popular with the development of the 5G network and automatic driving. This survey focuses on recent research advances in mmWave-based human sensing. This survey presents various hardware and the key techniques of mmWave sensing, reviews existing mmWave sensing works based on different sensing tasks, i.e., human tracking and localization, motion recognition, biometric measurement and human imaging. We further discuss potential challenges and future directions, including hardware and platforms, enhancing the applicability, novel sensing schemes, and integration with new mediums.

§ ACKNOWLEDGMENT

This work is supported by the National Science Fund of China under grant No. U21B2007, National Science Fund of China under grant No. 62202264 and the R&D Project of Key Core Technology and Generic Technology in Shanxi Province (2020XXX007).

[A]ADCAnalog-to-digital converter

[A]AiPAntenna in package
[A]AoAAngle of arrival

[A]ASLAmerican sign language
[A]ASRAuthentication success rate

[A]AUCArea under the ROC curve

[B]BANCbackground and noise cancellation

[C]CERCharacter error rate
[C]CFARConstant false alarm rate
[C]CGANConditional generative adversarial nets
[C]CIRChannel impulse response
[C]CNNConvolutional neural network
[C]COTSCommercial off-the-shelf
[C]CPDACrossover path disambiguation algorithm
[C]CPDPConcentrated position-Doppler profile

[D]DBSCANDensity-based spatial clustering of applications with noise
[D]DCTDiscrete cosine transformd
[D]DDBRDual-differential backgound removal
[D]DHHDeaf of hard-of-hearing
[D]DNNDeep neural network
[D]DoFDegrees of freedom

[E]EEREqual error rate
[E]EKFExtended Kalman filter
[E]EMDEmpirical mode decomposition algorithm

[F]FAFace authentication
[F]FFTFast fourier transform
[F]FMCWFrequency modulated continuous wave
[F]FSKFrequency shift keying

[G]GANGenerative adversarial network
[G]GRUGated recurrent network

[H]HARHuman activity recognition
[H]HMM-VMHidden markov model-based voting mechanism
[H]HRVHeart rate variability
[H]HWAHardware accelerator block

[I]IFIntermediate frequency
[I]IHTIterative hard thresholding
[I]IMFIntrinsic mode function
[I]IMUInertial measurement unit
[I]IRSIntelligent reflective surface
[I]ISACIntegrated sensing and communication

[J]JITJoint transmitter smoothing

[L]LSTMLong short term memory

[M]MCDMel-Cepstral distortion
[M]MFCCMel frequency cepstral coefficient

[M]MMSEMinimum mean square error

[M]MTIMoving target indication
[M]MUSICMultiple signal classification
[M]MVDRMinimum variance distortionless response

[P]PoIPoint of interest

[R]RAIRange angle image
[R]RDMRange-Doppler matrix
[R]RFRadio frequency

[R]RNNResidual neural network
[R]RPCCResidual phase cepstrum coefficients
[R]RSSReceiver signal strength
[R]RFICRadio-frequency integrated circuit

[S]SARSynthetic aperture radar
[S]SLAMSimultaneous localization and mapping

[S]SNRSignal-to-noise ratio

[S]STFTShort-time Fourier transform

[V]VCVVocal cord vibration
[V]VMDVariational modal decomposition

[W]WPTWavelet packet transform
[W]WERWord error rate

[]Jia Zhang
is currently a PhD. student in the School of Software and BNRist of Tsinghua University. He received his B.E. degree in Tsinghua University in 2019. His research interests include Internet of Things and wireless sensing.

[]Rui Xi is currently an Assistant Researcher at School of Computer Science and Engineering in University of Electronic Science Technology of China (UESTC) since August 2022. During 2019.07 - 2022.07, he was a postdoctoral researcher at Tsinghua University, worked with Prof. Yuan He. He obtained the M.S. and Ph.D. degrees in Computer Science from UESTC, Chengdu, China, in 2014 and 2019, respectively. His main research interests include Internet of Things, wireless sensing and AI.

[]Yuan He is an associate professor in the School of Software and BNRist of Tsinghua University. He received his B.E. degree in the University of Science and Technology of China, his M.E. degree in the Institute of Software, Chinese Academy of Sciences, and his PhD degree in Hong Kong University of Science and Technology. His research interests include wireless networks, Internet of Things, pervasive and mobile computing. He is a senior member of  IEEE and a member of ACM.

[]Yimiao Sun
is currently a PhD. student in Tsinghua University. He received his B.E. degree in the  University of Electronic Science and Technology of China (UESTC) in 2021. His research interests include Internet of Things and wireless sensing.

[]Xiuzhen Guo is a postdoc in the School of Software and BNRist of Tsinghua University. She received her B.E. degree in Southwest University, and her PhD degree in Tsinghua University. Her research interests include wireless networks and Internet of Things.

[]Weiguo Wang is currently a PhD. student in Tsinghua University. He received his B.E. degree in the University of Electronic Science and Technology of China (UESTC). His research interests include acoustic sensing and mobile computing.

[]Xin Na is currently a PhD. student in Tsinghua University. He received his B.E. degree in Tsinghua University in 2020. His research interests include wireless networks and Internet of Things.

[]Yunhao Liu received his B.S. degree in Automation Department from Tsinghua University, and an M.A. degree in Beijing Foreign Studies University, China. He received an M.S. and a Ph.D. degree in Computer Science and Engineering in Michigan State University, USA. He is now a professor at Automation Department and Dean of the GIX in Tsinghua University, China. He is a Fellow of IEEE and ACM.

[]Zhenguo Shi received his M.S.(in 2011) degree and PhD (in 2016) from Harbin Institute of Technology (HIT), China. He received his second PhD degree (in 2022) from University of Technology Sydney, Australia. He was a visiting student at the University of Leeds (2012-2013) and a visiting scholar at the University of Technology Sydney, Australia (2016- 2017). He is currently working as a Postdoctoral Research Associate in the School of Computing at the Macquarie University, Australia. His research interests include Wireless sensing, Human activity recognition, IoT, Deep learning, Cognitive radio, Interference alignment, Massive MIMO and wireless communication.

[]Tao Gu is currently a Professor at Macquarie University, and he is an expert in the fields of Internet of Things, Embedded/Edge AI, Mobile and Ubiquitous Computing. His publications typically appear in conferences including MobiCom, SenSys, IPSN, UbiComp and INFOCOM. He has also served a leadership role in many conferences, including General Co-Chair of MobiCom 2022 and TPC Co-Chair of IoTDI 2021.