VQ-DeepVSC: A Dual-Stage Vector Quantization Framework for Video Semantic Communication
Yongyi Miao Zhongdang Li Yang Wang Die Hu Jun Yan Youfang Wang














VQ-DeepVSC: A Dual-Stage Vector Quantization Framework for Video Semantic Communication
    Yongyi Miao+, Zhongdang Li+, Yang Wang, Die Hu*, Jun Yan, and Youfang Wang
        

Yongyi Miao, Zhongdang Li, Yang Wang, Die Hu, Jun Yan, and Youfang Wang are with the School of Information Science and Technology, Fudan University, Shanghai 200433, China.

    
======================================================================================================================================================================================================================================================================

§ ABSTRACT 
In response to the rapid growth of global video traffic and the limitations of traditional wireless transmission systems, we propose a novel dual-stage vector quantization framework, VQ-DeepVSC, tailored to enhance video transmission over wireless channels. 
In the first stage, we design the adaptive key-frame extractor and interpolator, deployed respectively at the transmitter and receiver, which intelligently select key frames to minimize inter-frame redundancy and mitigate the cliff-effect under challenging channel conditions. 
In the second stage, we propose the semantic vector quantization encoder and decoder, placed respectively at the transmitter and receiver, which efficiently compress key frames using advanced indexing and spatial normalization modules to reduce redundancy. 
Additionally, we propose adjustable index selection and recovery modules, enhancing compression efficiency and enabling flexible compression ratio adjustment.
Compared to the joint source-channel coding (JSCC) framework, the proposed framework exhibits superior compatibility with current digital communication systems.
Experimental results demonstrate that VQ-DeepVSC achieves substantial improvements in both Multi-Scale Structural Similarity (MS-SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) metrics than the H.265 standard, particularly under low channel signal-to-noise ratio (SNR) or multi-path channels, highlighting the significantly enhanced transmission capabilities of our approach.



Semantic Communication, Video Transmission, Vector Quantization, Multipath Fading Channel, Deep Learning.




§ INTRODUCTION

With the rapid advancement of information technology, global data traffic, especially video traffic, has grown exponentially, now constituting the predominant component of internet data traffic <cit.>. Despite enhancements in traditional wireless video transmission systems that focus on optimizing bit error rates (BER) <cit.>, challenges persist in ensuring high-quality transmission. These systems primarily emphasize compression efficiency but often fall short in addressing the semantic understanding and adaptability required for dynamic network conditions. Even with the adoption of the latest H.265 technology, the impact of the cliff-effect remains unresolved. This effect describes a significant decline in video transmission quality when the channel signal-to-noise ratio (SNR) drops below a critical threshold.



 §.§ Prior Work

The rapid development of deep learning (DL) technology has propelled semantic communication into a crucial role in the next generation of communication technologies, demonstrating significant potential and advantages <cit.>. Semantic communication, grounded in understanding before transmitting, involves extracting semantic information <cit.>. It enables profound compression by analyzing and distilling the essence of original content, proving highly adaptable and extensible across diverse domains, including text <cit.>, speech <cit.>, images <cit.>, and video <cit.>. Thus, it heralds a new era of intelligent and context-aware communication systems.

Despite considerable focus on the application of deep learning to video compression <cit.>, the advent of semantic-based deep learning to enhance wireless video transmission was marked by the introduction of DeepWiVe by Tung et al. in 2022 <cit.>. This pioneering approach presented an integrated solution for video compression, channel coding, and modulation through an end-to-end joint source-channel coding (JSCC) framework. Building upon this foundation, Zhang et al. further innovated by incorporating dual optical flow estimation for video transmission <cit.>. Dong et al. introduced Rosefinch, a semantic communication model for live media streaming, video conferencing, and low-rate image transmission <cit.>. With these developments, JSCC framework has become the main research paradigm for video semantic communication (VSC) systems.

Notably, Gong et al. <cit.> introduced an adaptive bit rate VSC system, modulating the bit rate based on network conditions. Niu et al. <cit.> added a channel and spatial attention mechanism to their VSC framework, enhancing adaptability. Bao et al. <cit.> proposed a model division VSC scheme to extract shared semantic features and counteract noisy channels. Liang et al. <cit.> developed the VISTA framework, using semantic location graphs to manage dynamic objects and adapt to changing channel conditions. 
Specialized systems have also been developed for specific scenarios. Jiang et al. <cit.> created a scalable video coding network for video conferencing, minimizing transmission resources. Liu et al. <cit.> introduced a federated learning-enhanced vehicle semantic communication framework, optimizing semantic extraction and resource allocation while maintaining data privacy. Wang et al. <cit.> employed a nonlinear transformation and conditional coding architecture to optimize the balance between transmission rate and distortion, underpinned by perceptual quality metrics. 


 §.§ Motivation and Contributions

While JSCC-based VSC systems have theoretically shown promise in reducing distortion and enhancing transmission efficiency, these systems map source data directly to channel symbols, causing constellation points to appear anywhere on the constellation diagram, which deviates from the design of current digital communication systems. 
In addition, most studies simulate channels based on additive white Gaussian noise (AWGN) or other simplistic models. Although the AWGN model is useful for controlled experiments, it fails to capture the complexities of real-world wireless communications, such as multipath effects, signal decay, and the ever-fluctuating nature of channel conditions. 
Therefore, the robustness and suitability of JSCC methodologies in complex, dynamic wireless environments require thorough validation.

To make semantic communication more compatible with digital communication systems, semantic communication systems for image transmission based on discrete latent spaces have been designed <cit.> <cit.> so that semantic features can be quantized into feature indices through the latent embedding space, thereby converting them into bitstreams. Subsequently, these bitstreams can be directly mapped into symbols for transmission using existing constellation mapping schemes.

The vector quantization (VQ) semantic communication system designed by Hu et al. <cit.>, Masked VQ-VAE, reduces transmitted information by quantizing and transmitting only feature indices. However, due to the limitations of VQ-VAE in image reconstruction <cit.>, it addresses downstream tasks solely by transmitting task-relevant key feature indices within the latent embedding space. In addition, VQ-DeepSC, designed by Fu et al. <cit.>, utilizes a U-Net structure <cit.> to extract features at different scales and the input image is mapped to single-channel indices at each scale. However, this method necessitates multiple latent embedding spaces of different sizes, leading to excessive storage requirements at both the transmitter and receiver ends. Furthermore, the information bottleneck problem within the U-Net structure limits its ability to capture remote global contextual information, thereby reducing its effectiveness in handling high-resolution images. Additionally, the quantization operator is lossy, and similar patches are often embedded with the same index, leading to the creation of pseudo-shadows and discontinuities in generated images. 

In addition to being limited by the image reconstruction capabilities of current semantic communication network structures, extending VQ-based semantic communication systems from image to video transmission is challenging. This is due to the unique demands of video transmission, such as the need to further reduce temporal redundancy.

In this paper, we firstly propose a dual-stage vector quantization-based video semantic communication system named VQ-DeepVSC. To enhance the physical interpretability of the system, VQ-DeepVSC is divided into two distinct stages to improve transmission efficiency, each specifically aimed at reducing redundancy in the temporal and spatial dimensions, respectively. 

In the first stage, we design an adaptive key-frame extraction and interpolation (AKEI) module to reduce temporal redundancy, transforming video transmission into a key-frame transmission task. Meanwhile, to mitigate the cliff-effect arising from the constrained channel coding capacity in separated source-channel coding systems, the proposed AKEI estimates frame importance and models the relationship between SNR and key-frame rate, prioritizing the retransmission of critical key frames under poor channel conditions (e.g., when SNR is  low). At the receiving end, an optical flow estimation-based interpolation algorithm restores the video to its original frame rate, enhancing robustness and stability in challenging channel conditions. 
In the second stage, we leverage a multi-channel spatially conditioned vector quantization (MSVQ) for the ultimate compression of key frames. Compared to existing methodologies such as Masked VQ-VAE <cit.> and VQ-DeepSC <cit.>, MSVQ utilizes a shared latent embedding space for multi-channel index mapping <cit.>, supporting high-resolution video transmission and promoting diversity without increasing latent space size. Spatial conditional normalization in the MSVQ decoder mitigates quantization artifacts, ensuring high-quality image reconstruction. Additionally, key-frame indices are further compressed using adjustable index selection and recovery algorithms, allowing flexible compression rates. The main contributions of this paper are summarized as follows:

    
  * We are the first to use DL-based VQ methods for video transmission. Compared to JSCC-based VSC systems, the VQ-DeepVSC adapts better to the variable conditions of real wireless channels and is more compatible with existing communication systems. The proposed MSVQ is designed to minimize intra-frame redundancies using multi-channel indexing and spatial conditional normalization, achieving a high compression degree while maintaining frame quality.
    
  * We design the AKEI module, a revolutionary framework that intelligently selects key frames based on content and wireless channel quality, thereby optimizing the semantic compression process, significantly reducing inter-frame redundancies, and effectively tackling the cliff-effect in video transmission.
    
  * We propose adjustable index selection and recovery algorithms that reduce redundancy between key frames by transmitting only indices significantly different from their predecessors, as determined by a preset threshold. This approach enhances compression efficiency, minimizes compression loss, and allows for more flexible adjustment of compression rates.
    
  * Extensive simulation experiments have validated that the proposed VQ-DeepVSC achieves superior reconstruction quality at equivalent compression rates compared to other methods. What's more, our system demonstrates exceptional robustness, maintaining high-performance transmission efficiency across various channel conditions.




 §.§ Organization

The structure of this paper is as follows: Section <ref> proposes the framework of the VQ-DeepVSC system. Section <ref> elaborates on the proposed AKEI. Section <ref> designs the implementation of the MSVQ. Section <ref> explores the strategy for reducing redundancy between key frames. Section <ref> presents experimental results that validate the performance of the proposed VQ-DeepVSC system. Finally, Section <ref> concludes the paper.

Notations: Superscripts ^T stands for transpose. 𝐈_𝑀×𝑁 represents the 𝑀×𝑁 identity matrix and 0_M× N denotes the M× N all-zero matrix. For a matrix 𝐀, [𝐀]_𝑖,: denotes the i-th row of 𝐀.
For a vector 𝐚, ||𝐚||_2 denotes the Euclidean norm, 𝐚(m) denotes the m-th elements of 𝐚, and 𝐚(𝑚:𝑛) represents taking the 𝑚-th to 𝑛-th elements from vector 𝐚. 
For a scalar x, ⌊ x ⌋ denotes the nearest integer smaller than or equal to x.
For operations, ⟨·, ·⟩ represents the inner product, ⊗ indicates the batch-wise matrix multiplication operation, and ⊙ denotes the Hadamard product.
Finally, ℝ^m × n and ℤ^m × n denote the spaces of m × n real and integer matrices, respectively.



§ FRAMEWORK OF VQ-BASED SEMANTIC COMMUNICATIONS FOR VIDEO TRANSMISSION
 

In this section, we introduce a novel framework for semantic communication systems specifically tailored for video transmission. This framework, based on VQ, 

is illustrated in Fig. <ref>. The transmitter consists of an adaptive key-frame extractor, a semantic vector quantization encoder, an adjustable index selector, channel coding and modulation, and orthogonal frequency division multiplexing (OFDM) modulation. The receiver comprises OFDM demodulation, channel estimation and channel equalization, a demodulation and channel decoder, an adjustable index restorer, a semantic vector quantization decoder, and an adaptive key-frame interpolator.


 §.§ Transmitter

As shown in Fig. <ref>, the video transmitter processes a sequence of video data, which spans an arbitrary duration T and is composed of N individual frames. Each frame 𝐅_n ∈ℝ^W_F× H_F× C_F with n = 1, …, N is an RGB image with the width of W_F, the height of H_F , and the C_F = 3 channels.

To avoid the cliff-effect, a feasible method is to retransmit video frames multiple times. However, this approach will undoubtedly increase the amount of transmitted data, greatly affecting the compression effect. To ensure that the compression ratio remains unchanged during retransmission, we propose an adaptive key-frame extractor that assesses the importance of each frame based on channel quality (such as SNR) and content changes, and extracts the key frames. Only the key frames are transmitted and retransmitted under poor channel conditions, and denote the m-th key frame as 𝐊_m ∈ℝ^W_F× H_F× C_F, where m = 1, …, M, and M represents the number of key frames, and we use the vector 𝐯∈ℤ^N × 1 to record the positions of the key frames. Specifically, if the 𝑛-th frame is the key frame, we set 𝐯(n)=1 , otherwise we set 𝐯(n)=0, where n = 1, …, N. The specific details of this process and its implications will be elaborated in Section <ref>.




We define the ratio of key frames to all frames ρ as:

    ρ = M/N,

which can be adaptively adjusted according to the channel conditions.

After extracting key frames, a semantic vector quantization encoder is utilized for feature extraction and indexing. Here, we utilize a latent embedding space which is denoted by 𝐄 = {𝐞_1, …, 𝐞_L}∈ℝ^L × d, where L denotes the size of 𝐄, d is the dimensionality of each latent embedding vector 𝐞_l in 𝐄, and l = 1, …, L. Space 𝐄 is trained during the training phase and is shared between the transmitter and receiver. Following feature extraction by the semantic vector quantization encoder, 𝑀 index sequences {𝐬_m}_m=1^M are generated, where 𝐬_m represents the index sequence of the m-th keyframe corresponding to the quantized vector in the 𝐄 space, m = 1, …, M. Details will be provided in Section <ref>. 

To further reduce redundancy, we propose an adjustable index selector that performs the final screening of the index sequence. The outputs from the adjustable index selector are the resulting indices 𝐬_η and the position sequence 𝐩. The specific details will be provided in Section <ref>.

After the aforementioned modules, the video data {𝐅_n}_n=1^N can ultimately be encoded into a bitstream 𝐛, which is composed of the following three parts, i.e.:

    𝐛 = [𝐛_𝐬, 𝐛_𝐩,𝐛_𝐯],

where 𝐛_𝐬, 𝐛_𝐩 and 𝐛_𝐯 correspond to the bitstreams of 𝐬_η, 𝐩 and 𝐯, respectively.


After obtaining 𝐛, operations identical to those used in traditional communication systems can be employed, namely encoding and modulating the bitstream for transmission. To combat multipath fading in the channel, OFDM technology is also utilized here. As can be seen from the figure, the semantic communication system we propose is fully compatible with existing communication systems.




 §.§ Receiver


The received time-domain signal at the receiver can be expressed as:

    y(t) = h(t,τ) * x(t) + n(t),

where h(t, τ) represents the multi-path time-varying channel impulse response, n(t) denotes the additive white Gaussian noise, and * signifies convolution.

After applying OFDM demodulation, channel equalization, demodulation, and channel decoding, we can 
finally obtain the estimates of 𝐬_η, 𝐩 and 𝐯, i.e., 𝐬̂_η, 𝐩̂ and 𝐯̂, respectively.

Subsequently, the adjustable index restorer is employed based on 𝐬̂_η and 𝐩̂, enabling us to retrieve {𝐬̂_m}_m=1^M.
Given 𝐯̂, the semantic vector quantization decoder is then used to recover the key frames, i.e., to obtain {𝐊̂_m}_m=1^M. Finally, given {𝐊̂_m}_m=1^M and 𝐯̂, we use an adaptive key-frame interpolator to recover all video frames {𝐅̂_n}_n=1^N. 




§ THE PROPOSED AKEI


Many frames within a video sequence are often similar, containing redundant information. Therefore, the transmission of data can be reduced by only transmitting a subset of frames and then performing frame interpolation at the receiver. These transmitted frames are known as key frames. Here, we propose AKEI to achieve the extraction of key frames and the subsequent recovery of all frames.

AKEI aims to reduce inter-frame redundancies in video transmission and address cliff-effect. AKEI consists of two primary components: the adaptive key-frame extractor, located on the transmitter, and the adaptive key-frame interpolator, situated on the receiver. 

When the SNR decreases, the quality of video transmission may suddenly drop at a certain point, a phenomenon known as the cliff-effect. An effective method to address the cliff-effect is to perform multiple retransmissions of the video, but this can lead to a significant decrease in compression capability of the system. To solve the cliff-effect without increasing the compression rate, we employ an adaptive key-frame extractor to dynamically select key frames for transmission and perform multiple retransmissions under low SNRs. The adaptive key-frame extractor dynamically selects key frames and generates a vector to record the positions of these key frames within the video sequence. This selection process is adaptively adjusted based on content variation between video frames and the quality of the multi-path channel. 

Conversely, the adaptive key-frame interpolator utilizes the received key frames to reconstruct the full video sequence. To facilitate practical deployment, AKEI is designed with an asymmetric structure, featuring a simpler structure at the receiver.







 §.§ Adaptive Key-frame Extractor


As illustrated in Fig. <ref>, the input of the adaptive key-frame extractor is {𝐅_n}_n=1^N. And the output is the key frames {𝐊_m}_m=1^M. Initially, an incremental flow network (IFNet) is used to estimate the optical flow of frames and outputs {𝐎_i}_i=2^N-1. Each optical flow map 𝐎_i ∈ℝ^C_O× H_O× W_O denotes the optical flow corresponding to frame 𝐅_i, where C_O, H_O, and W_O represent the number of channels, height, and width of 𝐎_i, respectively. These dimensions are consistent across all 𝐎_i.
By default, we assume that 𝐅_1 and 𝐅_N are the key frames, i.e., we set 𝐯(1) = 1 and 𝐯(N) = 1. Therefore, the IFNet only computes the optical flow for the intermediate (N-2) frames. Unlike dual optical flow networks that provide two optical flow estimations for each frame to represent the content changes relative to the preceding and succeeding frames, our approach only outputs one optical flow estimation for each frame, indicating the content changes relative to the surrounding frames. 


Suppose that the frame to be reconstructed is 𝐅_i, where i = 2, …, N-1, the FusionNet utilizes its preceding frame 𝐅_i-1 and succeeding frame 𝐅_i+1, as well as the optical flow estimation 𝐎_i of 𝐅_i, to reconstruct 𝐅_i. Here, we denote the reconstruction result by 𝐅̌_i.


Following FusionNet, the frame importance score (FIS) calculator outputs the score sequence β∈ℝ^N × 1. These scores, derived from comparisons between {𝐅̌_n}_n=2^N-1 and {𝐅_n}_n=2^N-1, are assigned to β(2:N-1). β(1) and β(N) are set to sufficiently large values.
In the end, the mask generator determines which frames among the N frames are key frames based on the β and channel conditions (e.g., SNR), and outputs 𝐯. 

The details of the four modules in Fig. <ref> are as follows:


  §.§.§ IFNet


IFNet diverges from traditional dual optical flow networks by adopting a coarse-to-fine strategy that directly estimates optical flow for intermediate frames. Simple linear interpolation of forward and backward flows may falter in the presence of rapid motion or complex backgrounds. In contrast, our approach mitigates artifacts and blurriness by iteratively refining optical flow estimation. The specific structure of the IFNet comprises multiple incremental flow blocks (IFBlocks). IFNet utilizes the preceding and succeeding frames of the current frame, namely 𝐅_pre and 𝐅_next, to calculate the optical flow 𝐎_cur of the current frame. IFNet employs a coarse-to-fine strategy through three IFBlocks to progressively enhance the accuracy of optical flow estimation. Each IFBlock utilizes a distinct downsampling factor α_i (set as 4, 2, 1 respectively) to reduce the spatial dimensions of the image and extracts features using convolutional 2D layers, denoted as Conv2d. As resolution is incrementally enhanced, finer details are captured. Finally, upsampling restores the image to its original resolution through deconvlutional 2d layers, coupled with Bilinear Resize operations, achieving high-precision optical flow estimation.

This coarse-to-fine approach iteratively corrects motion estimation inaccuracies. By refining optical flow across stages, our method effectively reduces artifacts and blurriness, resulting in more precise and higher-quality interpolation of intermediate frames.


  §.§.§ FusionNet

FusionNet utilizes two ContextNets to extract background features from the video frames 𝐅_pre and 𝐅_next, and considers the results of optical flow estimation through WarpBlock operations, ultimately obtaining 𝐎^1,2,3,4_pre and 𝐎^1,2,3,4_next, respectively. ContextNet consists of four WarpBlocks. The first WarpBlock accepts the optical flow results generated by IFNet, along with either the previous frame 𝐅_pre or the next frame 𝐅_next as input. Subsequent WarpBlocks, specifically the i-th WarpBlock with i=2,3,4, take as inputs the intermediate results 𝐎^i-1 and 𝐅^i-1, which are produced by the (i-1)-th WarpBlock. The i-th WarpBlock utilizes Conv2d to extract features from 𝐎^𝑖-1, and employs Bilinear Resize to restore the size of 𝐎^𝑖 to obtain 𝐎^𝑖. Concurrently, 𝐎^𝑖 is warped with 𝐅^𝑖-1 to yield 𝐅^𝑖. Each WarpBlock allows for the learning of background information to varying degrees. Then a U-Net structured network is employed to continually perform Conv2d and supplement 𝐎^i to extract multi-scale features, and uses Deconv2d to perceive and reconstruct these features, ultimately yielding the frame reconstruction result.




  §.§.§ Frame Importance Score Calculator

To assess the importance of frames based on video content changes and motion information, we develop a new evaluative metric called the frame importance score, denoted by β, which  can reflect the degree of difference between the reconstructed frames {𝐅̌_n}_n=2^N-1 and the original frames {𝐅_n}_n=2^N-1. A higher value in β indicates a greater distinction between the two images, signifying that the frame is more challenging to reconstruct, and thus, this frame is deemed more significant. The score of the n-th frame can be given by:

    β(n) = f_is(𝐅̌_n,𝐅_n),

where f_is(· , · ) represents the assessment function, which is contingent upon the specific application context. 
For instance, one can calculate the Structural Similarity (SSIM) <cit.> or the Learned Perceptual Image Patch Similarity (LPIPS) <cit.> between 𝐅̌_n and 𝐅_n. 




  §.§.§ Adaptive Frame Selection


Before selecting key frames, it is essential to determine the key frames ratio ρ.
When the channel condition is poor, it is essential to reduct transmission resources for retransmitting the most critical key frames, specifically those with higher β values. This adjustment effectively mitigates the cliff-effect, which deteriorates video quality significantly under extremely poor channel conditions.
By conserving bandwidth through prioritizing critical frames, the strategy ensures reliable reception of essential information even in challenging channel conditions.

Here, we adopt SNR, denoted by   γ, to represent channel conditions because of its simplicity and ease of acquisition in practice. We establish a quantitative relationship between γ and ρ as follows:

    ρ = ∑_i=0^I a_i · (log(γ))^i,

where ρ represents the predicted key-frame ratio based on γ, a_i are the coefficients of the polynomial regression model, and I is the degree of the polynomial.

The polynomial coefficients 𝐚 = [a_0, a_1, …, a_I] can be determined by minimizing the objective function Err, which measures the discrepancy between observed and model-predicted key-frame ratios:

    𝐚 = min_𝐚{ Err + υ· R(𝐚) }.

Here, υ is a custom parameter that adjusts the balance between the data fit and the regularization term, and R(𝐚) is the regularization function given by:

    R(𝐚) =
    𝐚_1 = ∑_i=0^I |a_i|,    for L1 regularization, 
    𝐚_2^2 = ∑_i=0^I a_i^2,    for L2 regularization.

Here, R(𝐚) applies either L1 or L2 regularization to the coefficients 𝐚, with L1 promoting sparsity and L2 encouraging smaller values across the coefficients. Err is defined as:

    Err = ( ρ - ( ∑_i=0^I a_i · (log(γ))^i ) )^2,

which quantifies the squared difference between the actual key-frame ratio and that predicted by the model.


Given γ, we can obtain ρ and then M. Based on M and β, we can determine the key-frames by selecting the frames corresponding to the top M values in β. These frames are then marked as key-frames by setting their positions in the sequence of N frames to 1.


 §.§ Adaptive Key-frame Interpolator


The adaptive key-frame interpolator is designed with a simpler structure than the transmitter's adaptive key-frame extractor to accommodate the practical conditions of wireless communication. As illustrated in Fig. <ref>, its inputs are {𝐊̂_m}_m=1^M and 𝐯̂, which are the estimates of {𝐊_m}_m=1^M and 𝐯, repectively. Initially, the gap calculator generates a sequence 𝐠∈ℤ^(M-1) × 1, which records the number of non-key frames between each pair of consecutive key frames based on 𝐯̂. Subsequently, all key frames along with 𝐠 are input into the IFNet to calculate the optical flow {𝐎̂_i}_i=1^N-M of the  (N-M) non-key frames interspersed between the key frames. The FusionNet then reconstructs the non-key frames {𝐊̃_i }_i=1^N-M, by utilizing the information from {𝐎̂_i }_i=1^N-M. Finally, we use the video seam module to stitch together {𝐊̃_i }_i=1^N-M and {𝐊̂_m }_m=1^M according to 𝐠, reconstructing the sequence of video frames {𝐅̂_n }_n=1^N.





§ THE PROPOSED MSVQ




To optimize compression efficiency for each key-frame, we propose MSVQ that can reduce intra-frame redundancies. MSVQ module consists of two key components: the semantic vector quantization encoder at the transmitter and the semantic vector quantization decoder at the receiver. The former possesses exceptional image compression capability, encoding the frames into multi-channel indices, while the latter accurately reconstructs the original frames from these indices.






 §.§ Semantic Vector Quantization Encoder



  §.§.§ Procedure

As shown in Fig. <ref>, the encoder first applies a CNN encoder to extract multi-channel features 𝐳_m= [𝐳_m^(1),…,𝐳_m^(c)] ∈ℝ^h× w× d× c from the m-th key-frame 𝐊_m where 𝐳_m^(j)∈ℝ^h× w× d denotes the j-th channel feature of the m-th frame, j=1,2,… ,c, and c represents the number of channels. 

The feature 𝐳_m^(j) is reshaped to 𝐳̅_m^(j)∈ℝ^U × d, where U = hw and its u-th row, i.e., [𝐳̅_m^(j)]_u,:, is then quantized to the index of the nearest vector in the latent embedding space 𝐄, i.e.,

    𝐬̅_m^(j)(u) = largmin[𝐳̅_m^(j)]_u,: - 𝐞_l_2,

where l = 1, …, L and 𝐬̅_m^(j)∈ℤ^U × 1. Once L is given, each index value, i.e., 𝐬̅_m^(j)(u), can be represented by B = log_2 L bits.

Subsequently, we concatenate these index sequences to form 𝐬_m = [𝐬̅_m^(1); …; 𝐬̅_m^(c)] ∈ℤ^L_𝐬× 1, where L_𝐬=Uc. Thus, employing the semantic vector quantization encoder compresses the data of a video frame into a vector of length L_𝐬, equivalent to a bitstream of length L_𝐬B.






  §.§.§ Details of CNN Encoder

The CNN encoder in MSVQ is capable of encoding the input into a multi-channel vector rather than a single channel. The advantages of multi-channel representation over single-channel representation include increased information capacity, enhanced feature combinations, and improved image generation quality <cit.>. 

Our CNN encoder incorporates multi-level downsampling modules to extract multi-scale features and expand the receptive field of each feature point, thus improving the model's ability to capture global structure and semantic information from the input data. Each down-sample module consists of two residual blocks, which help mitigate the vanishing gradient problem, facilitate the training of deeper networks, and improve overall network performance. To assist the network in better understanding more abstract and complex patterns that may exist in the deep features, an attention mechanism module is added in the last down-sample module. 
Finally, the features are mapped to the latent embedding space through an intermediate layer and various operations.







 §.§ Semantic Vector Quantization Decoder



  §.§.§ Procedure

The semantic vector quantization decoder depicted in Fig. <ref> takes the index sequence 𝐬̂_m as input. This sequence is first mapped back to the corresponding vector 𝐳̂_m in the latent embedding space. Subsequently, it is decoded using a CNN decoder to reconstruct the key-frame 𝐊_m. Finally, a patch-based discriminator 𝒟 is employed to assess the constructed frame 𝐊̂_m. It should be pointed out that during the training phase of this model, we directly assign 𝐬̂_m = 𝐬_m.



  §.§.§ Details of CNN Decoder and Discriminator

Compared to the CNN encoder, the CNN decoder in MSVQ exhibits a mirrored architecture in certain aspects. The decoder includes an intermediate layer, multiple up-sample modules, and concludes with a final Norm, SiLU, and Conv2d. The structures of the residual block and the attention mechanism module are consistent with those in the CNN encoder.

However, the CNN decoder is not completely symmetrical to the CNN encoder. It is noteworthy that all Norms in the CNN decoder, including residual blocks and attention module blocks, are replaced by spatial conditional normalization (SCN). This ensures that the same quantized indices produce results that are not identical but more natural at different positions, thereby reducing artifacts and discontinuities in the generated frames compared to conventional DL-based VQ methods <cit.>. 

The SCN module initially standardizes the intermediate feature map 𝐟^i∈ℝ^C_𝐟^i× H_𝐟^i× W_𝐟^i via group normalization to eliminate stylistic variations, where C_𝐟^i represents the number of channels, while H_𝐟^i and W_𝐟^i represent the height and width of the feature map respectively. 
Then, it integrates the embedded vectors 𝐳̂_m as auxiliary information into the input feature maps. Specifically, this is achieved using (<ref>):


    𝐟^i+1 = 𝐟^i - μ_GN(𝐟^i)/σ_GN(𝐟^i)⊙Θ_y(Φ(𝐳̂_m)) + Θ_b(Φ(𝐳̂_m)),

where function Φ(·) interpolates and adjusts the auxiliary input 𝐳̂_m to match the size of the input feature map 𝐟^i, Φ(𝐳̂_m)∈ℝ^ c × H_𝐟^i× W_𝐟^i and c represents the number of channels described in Section <ref>. Θ_y and Θ_b are two learnable affine transformations that map the adjusted 𝐳̂_m to the same number of channels as 𝐟^i. Additionally, μ_GN and σ_GN respectively represent the mean and standard deviation of the group normalization applied to 𝐟^i.
In summary, the normalized feature map is multiplied and added with the convolutionally mapped auxiliary information to generate the final output feature map 𝐟^i+1.





 §.§ Loss Function

During the training phase of the encoder, the loss function we use consists of three components: reconstruction loss, vector quantization loss, and adversarial loss. 

The reconstruction loss measures the difference between the frames generated by the model and the original ones, which is given by: 

    ℒ_rec(𝐊_m,𝐊̂_m) = 𝐊_m - 𝐊̂_m_2^2 + λℒ_perceptual(𝐊_m, 𝐊̂_m),

where λ is the weight coefficient.
ℒ_perceptual quantifies the perceptual distance between the generated frames and the original frames by computing the distance between the feature maps extracted from the generated and input frames through networks. In this paper, the feature outputs of five convolutional layers in the VGG-16 model <cit.> are selected to compute the perceptual loss as:

    ℒ_perceptual(𝐊_m,𝐊̂_m)=∑_j=0^41/C_jH_jW_j∥φ_j(𝐊_m)-φ_j(𝐊̂_m)∥_2^2,

where φ_j is the output of the j-th layer of the feature extraction network, and C_j, H_j, W_j represent the number of channels, height, and width of the feature maps at the j-th layer, respectively.

The vector quantization loss is incurred during the quantization process due to the straight-through gradient estimator method, which can be calculated as:

    ℒ_VQ(𝐳_m,𝐳̂_m) =sg[𝐳_m]-𝐳̂_m_2^2+sg[𝐳̂_m]-𝐳_m_2^2.

Here sg[ ·  ] denotes the stop-gradient operation.

The adversarial loss is given by: 

    ℒ_GAN(𝐊_m,𝐊̂_m;θ)=log𝒟(𝐊_m;θ)+log(1-𝒟(𝐊̂_m;θ)),

where θ represents learnable parameters in 𝒟.

The whole objective function for identifying the optimal model can be expressed as:

    𝒬^*    = 𝐰_𝖤, 𝐰_𝖦, 𝐰_𝒵*minθ*max𝔼_𝐊_m ∼ p(𝐊_m)[ ℒ_rec + μ_1 ℒ_VQ + μ_2 ℒ_GAN],

where 𝐰_𝖤, 𝐰_𝖦, and 𝐰_𝒵 denote the learnable parameters of the CNN encoder, the CNN decoder, and the latent embedding space, respectively. μ_1 and μ_2 are weighting coefficients that adjust the contributions of the vector quantization loss and the generative adversarial network loss to the total loss. 𝔼_𝐊_m ∼ p(𝐊_m) represents the expected value over the distribution p(𝐊_m) of the input frame 𝐊_m.




§ ADJUSTABLE INDEX SELECTOR AND RESTORER

Using the semantic vector quantization encoder, the m-th key frame is represented by the sequence of indices 𝐬_m. 
Letting 𝐪_m ∈ℝ^L_𝐬× d represent the matrix that consists of L_𝐬 quantized feature vectors for the m-th key frame, we have 

    [𝐪_m]_i,: = 𝐞_𝐬_m(i)^T ,

where i = 1, 2, …, L_𝐬 and 𝐞_𝐬_m(i) refers to the 𝐬_m(i)-th basis vector in the latent embedding space 𝐄.

In practice, the quantized feature vectors between adjacent key frames may exhibit similarity. By computing the similarity between the feature vectors of the 𝐊_m-1 and 𝐊_m, we can assess the redundancy between consecutive frames.

Take cosine similarity as an example, which is given by:

    𝑠𝑖𝑚 = ⟨[𝐪_m-1]_i,:, [𝐪_m]_i,:⟩/[𝐪_m-1]_i,:_2 ·[𝐪_m]_i,:_2.

If sim exceeds a predefined threshold η, it indicates substantial redundancy between the two frames. In such cases, transmission can be optimized by selectively transmitting only the indices that exhibit significant changes, thereby reducing the volume of data transmitted while minimizing the impact on video reconstruction quality. 

Let 𝐬_η denote the final sequence of selected indices and define:

    𝐩 = [𝐩_2; …; 𝐩_M] ,

where 𝐩_m is a length L_𝐬 vector whose element is 0 or 1, and m = 2, …, M. Specifically, if the cosine similarity between [𝐪_m-1]_i,: and [𝐪_m]_i,: is less than η, we set 𝐩_m(i)=1 and include 𝐬_m(i) in 𝐬_η. Otherwise, we set 𝐩_m(i)=0 and keep 𝐬_η unchanged.

Upon receiving the estimates of  𝐬_η and 𝐩, i.e., 𝐬̂_η and 𝐩̂ at the receiver, the adjustable index restorer reconstructs the key frame index sequence {𝐬̂_m}_m=1^M. Initially, 𝐬̂_1 is initialized using 𝐬̂_η.
Then for each subsequent key frame m = 2,…,M, 𝐬̂_m is reconstructed based on 𝐬̂_m-1 and using 𝐬̂_η with consideration of 𝐩̂. 

As shown in Fig. <ref>, after processing the original video frames {𝐅_n}_n=1^N through the adaptive key-frame extractor, semantic vector quantization encoder, and adjustable index selector modules, we can finally obtain 𝐯, 𝐬_η, and 𝐩. Define the bit compression ratio (BCR) as the ratio of the number of bits required for the final transmission sequence to the original video frames. Thus, the BCR of the proposed VQ-DeepVSC is given by:

    BCR = N + L_η× B + L_𝐬× (M-1)/C_F× H_F× W_F× 8 × N,

where L_η is the length of 𝐬_η, and N, L_ηB, and L_𝐬(M-1) are the number of bits required for 𝐯, 𝐬_η, and 𝐩, respectively.



§ EXPERIMENTS

In this section, we provide a detailed description of the training configurations for the AKEI and MSVQ stages, as well as the conditions for the transmission experiments. We focus on the significance of AKEI and the adjustable index selector and restorer and their impact on system performance. Finally, we compare VQ-DeepVSC with the traditional H.265 method through experiments.




 §.§ Implementation Details

In simulations, we employ low-density parity-check code (LDPC) codes with a block length of 648 bits and a 3/4 rate to encode 𝐛. Both quadrature phase shift keying (QPSK) and 16-quadrature amplitude modulation (16QAM)  signaling schemes are utilized. 


The subsequent experimental evaluation assesses the performance of the proposed VQ-DeepVSC using the UCF101 <cit.> dataset. The UCF101 dataset, comprising 101 distinct video categories, allows for a thorough examination of the system's  capabilities. From each category, we randomly select a subset of 10 videos, creating a test dataset of 1,010 videos. This diverse test set allows for a rigorous evaluation of the efficacy and reliability of VQ-DeepVSC in video transmission, ensuring a more robust assessment of the performance of the different modules within the system.



To evaluate the system performance, we employ MS-SSIM <cit.> to compare the multiscale structural similarity at the pixel level between each frame of the original video and its reconstructed counterpart, and utilize LPIPS <cit.> to assess the perceptual quality of the reconstructed video from the human perspective.







 §.§ Comparative Analysis of Methods


We compare the performance of VQ-DeepVSC with typical and commonly used H.265 video compression codecs for video transmission. We utilize the FFMPEG <cit.> library to perform H.265 encoding and decoding. The BCR of H.265 is given by:

    BCR = B_s× 8/C_F× H_F× W_F× 8 × N,

where B_s represents the file size in bytes after H.265 compression. 
According to (<ref>), the BCR for H.265 on the test dataset is 0.024.
For VQ-DeepVSC, we set η to 1, and based on (<ref>), we can obtain the BCR of our system is 0.023, which is slightly lower than that of H.265.

Fig. <ref> and Fig. <ref> show the comparisons between the proposed VQ-DeepVSC and H.265 video codecs under AWGN and multipath fading channel, respectively. 
From the figures, it can be observed that the proposed VQ-DeepVSC outperforms H.265 in terms of both MS-SSIM and LPIPS, especially at medium to low SNRs. 
This is due to H.265 being sensitive to noise, thereby limiting its effectiveness at medium to low SNRs. In contrast, our method not only achieves reliable video transmission at low SNRs but also maintains high video quality. 
It can also be observed that under multipath fading channel, H.265 fails to transmit videos effectively, whereas the proposed method maintains robust video transmission and ensures high quality. 



§ CONCLUSIONS

In this paper, we present the VQ-DeepVSC system, which employs an innovative dual-stage vector quantization framework to enhance video transmission quality and efficiency over wireless channels. This system effectively mitigates the cliff-effect and maintains significant data compression even under low SNR conditions, ensuring robust video transmission.
The proposed VQ-DeepVSC system employs the AKEI module in the first stage to extract key frames, reducing inter-frame redundancy and enhancing transmission robustness under low SNR conditions.
In the second stage, the MSVQ module compresses these key frames using a shared latent embedding space, effectively reducing intra-frame redundancy and supporting high-resolution video transmission. Additionally, the adjustable index selector and restorer further reduce inter-frame redundancy by compressing key-frame indices, enabling more flexible compression rates.
Experimental results demonstrate that the proposed VQ-DeepVSC achieves higher compression degree and higher video quality than the H.265 standard, especially under low SNRs or multi-path channels. Simulation results also demonstrate that the proposed VQ-DeepVSC has excellent generalization capabilities, making it suitable for various types of video transmission.

ieeetr



