FrameCorr: Adaptive, Autoencoder-based Neural Compression for Video Reconstruction in Resource and Timing Constrained Network Settings
John Li Shehab Sarar Ahmed Deepak Nair

Certifying Quantum Temporal Correlation via Randomized Measurements: Theory and Experiment
    Dawei Lu
     
==========================================================================================



2
§ ABSTRACT 
Despite the growing adoption of video processing via Internet of Things (IoT) devices due to their cost-effectiveness, transmitting captured data to nearby servers poses challenges due to varying timing constraints and scarcity of network bandwidth. Existing video compression methods face difficulties in recovering compressed data when incomplete data is provided. Here, we introduce , a deep-learning based solution that utilizes previously received data to predict the missing segments of a frame, enabling the reconstruction of a frame from partially received data.



Video Transmission,
Progressive Compression,
IoT



§ INTRODUCTION


Video-enabled IoT devices provide a comprehensive view of the environment by capturing visual data alongside traditional sensor data, facilitating real-time monitoring and decision-making across various domains. However, due to resource constraints, these devices often rely on edge servers for video processing. This reliance introduces timing constraints that may require interrupting frame transmission to transition to the next frame. Consequently, edge servers frequently face the challenge of reconstructing video frames from incomplete data. Thus, there is a pressing need for an efficient method on the server side to effectively handle missing data in video frames.

One way of handling this challenge is to encode each frame of the video into a compressed form before transmission.
There has been numerous compression techniques, both classical (Huffman Coding <cit.>, JPEG <cit.>, MPEG <cit.>, H.261 <cit.>, H.263 <cit.>, H.264 <cit.>, HEVC <cit.>) and neural network based ones using multilayer perceptrons (MLPs) <cit.>, Convolutional Neural Networks (CNNs) <cit.> and AutoEncoders <cit.>, which reduce overall frame size.
However, none of these methods is explicitly designed to manage decompression with partially received data, which often becomes the only recourse when the sender is unable to transmit complete data due to shortages in time.

In one very recent method, Progressive Neural Compression (PNC)  <cit.>, the authors propose a progressive encoding of images that can tolerate missing data.
However, it only applies to static images and relies on zero-filling to address missing data, a method that may result in suboptimal performance for videos due to not leveraging the inter-frame correlation between consecutive frames.

This paper presents , a deep-learning framework designed to exploit inter-frame correlation for efficiently reconstructing missing data within a frame.
Additionally, we implemented our own version of adaptive bitrate (ABR) video delivery on top of AVC to juxtapose its performance with that of , aiming to highlight differences in their methodologies. Unlike , which involves partitioning and extracting image frames and features from the videos, ABR solely adheres to the same video format without such extractions. We observed that AVC, when paired with ABR, outperforms deep learning-based methods in terms of both throughput and accuracy. Nonetheless, traditional algorithms like AVC exhibit limitations when confronted with incomplete data, rendering them unsuitable for tasks with strict timing requirements.





§ RELATED WORK

Conventional Image and Video Compression. Traditional methods of compression are primarily designed to reduce data volume necessary for image restoration. There are lossy compression techniques such as the coefficient quantization step in JPEG's 3-stage compression algorithm and lossless methods like discrete cosine transform (DCT) transformation in the same JPEG compression process <cit.>. Furthermore, JPEG has a progressive mode, allowing images to be compressed and reconstructed more flexibly based on an arbitrary amount of encoded data, where more data produces a more accurate reconstruction <cit.>.

However, such compression algorithms are only applicable to static images. Video content is addressed through other compression protocols. Low latency video coding and compression is especially relevant in IoT/Edge computing systems due to the real time deadlines that these applications generally operate under. One such video compression method is the H.264 <cit.> codec, also referred to as AVC, a version of the MPEG standard that incorporates block-based motion compensation strategies and exploits spatial and temporal repetitions. H.264 uses a combination of I-frames (Intra frames: these are complete frames that contain full image information) and P-frames (Predictive frames: these encode the differences or changes between themselves and previous reference frames) for video compression. I-frames are periodically inserted in the video stream or at scene changes to provide starting points for decoding. P-frames exploit spatial and temporal redundancies by “predicting" image content based on previously decoded frames.

Adaptive Video Compression
Adaptive video compression improves upon traditional methods by adjusting compression levels based on network conditions before transmission. A prominent example is adaptive bitrate (ABR) streaming, where video streams are compressed at various bitrates. Then, depending on the current network conditions, the appropriate bitrate-encoded videos are transmitted. Specific ABR protocols include HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH) <cit.>.

Neural Compression and Inference.
The advent of deep learning has ushered in innovative mechanisms for image compression, such as autoencoders (AEs) <cit.>, nested quantization, latent ordering <cit.>, and recurrent neural networks (RNNs) <cit.>. These methods enable variable compression, allowing for incremental refinement of image quality. Despite their superior compression efficacy, these neural models often demand significant computational resources, sometimes requiring several hours on GPU clusters during training, making them impractical for resource-limited IoT and edge devices.

Progressive Neural Compression.
Neural compression techniques have evolved to include progressive compression strategies, essential for adapting to fluctuating bandwidth conditions common in wireless sensor networks and distributed IoT applications. One such recent approach, Starfish <cit.>, introduces a method to enhance the resilience of neural compression to data transmission losses by adding random dropouts to its AE's bottleneck layer. Though Starfish does mitigate the impact of data loss, it lacks a mechanism for assessing and prioritizing the encoded features based on their importance for inference accuracy.

As aforementioned, PNC <cit.> was developed to improve classification accuracy for images within edge offloading environments, particularly when faced with temporal and bandwidth limitations. Diverging from existing methodologies, PNC dynamically adjusts to changes in bandwidth, allowing for efficient image classification by the edge server. It does so by training a multi-objective rateless autoencoder, tailored for multiple compression rates. PNC also implements a stochastic taildrop algorithm during training to form a compression solution that creates features ordered by importance in the inference process.
However, PNC is designed to work with static images, hence does not leverage the inherent correlation between video frames in the process of filling up the missing data.



§ SYSTEM MODEL

Our system architecture, as depicted in Figure <ref>, includes a resource-constrained system (e.g., an IoT device or low-power virtual machine), responsible for transmitting compressed video data. This data is then sent over a wireless network to a central edge server. At the server, a decompression algorithm—such as PNC or —is utilized to reconstruct the received compressed data (usually in bytes) into their original video or image frame formats as accurately as possible.

An essential consideration in the transmission of compressed frames is adherence to strict deadlines at the sender's end. Due to potentially limited network bandwidth, the client device may still be sending a frame when the subsequent frame becomes ready for transmission. Consequently, we enforce a deadline for each frame's transmission, requiring the transmission of the next frame commence promptly, even if a portion of the current frame remains unsent. As a result, the receiving server must be able to robustly reconstruct frames based on partially received data, a key component of . 







§ METHODS



 §.§ Dataset

Our dataset was derived from the UCF Sports Action dataset <cit.>, which originally comprises videos showcasing 10 distinct actions. From this dataset, we selected 8 of the 10 actions. Subsequently, we partitioned the videos from each action into training, validation, and test sets. The distribution of videos across each category is detailed in Table <ref>.



 §.§ AVC


Initially, we selected the AVC/H.264 codec as our baseline for video compression due to its widespread use in numerous applications, minimal data loss, and ability to maintain the original video format without the need for additional frame conversion. However, it exhibits limited error-resilience, especially in network environments with unstable bandwidth or strict timing constraints. Furthermore, AVC mandates transmitting the entire encoded video over the network; dynamic transmission of partial video segments isn't possible. 


For our experiments, we utilized the FFmpeg library with .mp4 as the video format. 



 §.§ ABR


To expand on the baseline H.264 method, we developed our own adaptive bitrate (ABR) video transmission implementation, which utilizes H.264 as the base encoding method and encodes content at various bitrates, specifically tweaking the control rate factor (CRF). In the FFmpeg library, a lower CRF (e.g. CRF=18) denotes a higher bitrate while a higher CRF (e.g. CRF=30) indicates a lower bitrate.



 §.§ PNC

PNC, as described in <cit.>, is a progressive encoding framework primarily designed for images. The PNC model undergoes a two-step training process. Initially, an autoencoder is trained to reconstruct images with high fidelity. Subsequently, the autoencoder is fine-tuned to optimize the accuracy of an image classifier using the reconstructed images. Throughout the training process, a stochastic tail-drop technique is employed to enhance the autoencoder's ability to reconstruct images from partially received data. In this technique, missing data is padded with zeros, ensuring that the decoder receives a fixed-size vector.

In our context, our focus is precisely on the reconstruction of video frames. Thus, we tailor the training of the autoencoder to prioritize minimizing the reconstruction error of these frames. To formalize this, let x_i denote the i^th captured frame, and E(x_i)=c_i represent the encoded data of x_i, computed by the IoT device. These encoded representations, denoted as c_i, are transmitted over the network. However, due to timing constraints, the sender may opt to switch to encoding and sending the next frame, thereby interrupting the transmission of the current frame. Consequently, the received data for the i^th frame is denoted as ċ_i. Upon receiving ċ_i, the receiver zero-pads it to match the dimension of c_i, resulting in ĉ_i. Subsequently, ĉ_i is passed through the decoder to reconstruct the frame, denoted as x̂_i. PNC is trained to minimize the mean squared error (MSE) between the original frame (x_i) and its reconstructed counterpart (x̂_i).

In conclusion, although consecutive frames within a video usually exhibit correlation, PNC, being originally designed for images, does not leverage this correlation. Consequently, missing data is filled with zeros during reconstruction.





 §.§ 

As previously mentioned, given the inherent correlation present in consecutive frames, our aim is to enhance PNC by leveraging this relationship to fill in missing data rather than simply padding it with zeros. 

This is where  comes into play. 
The architecture of  along with the PNC autoencoder is illustrated in Figure <ref>. The  model takes as input the encoded information from the preceding K frames (ĉ_̂ĵ's). It then predicts the encoded information for the current frame (c̃_̃ĩ). To address the possibility of missing data, stochastic taildrop is applied to the ĉ_̂ĵ's in the training phase. When there is a missing segment in the received data, the missing parts are filled up to produce ĉ_̂î, using the predicted value from the output of 's decoder component.










§ RESULTS



 §.§ Experimental Setup

Our experimental setup utilized two virtual machines (VMs) within a remote cluster farm, modeling the IoT device and edge server in our system. Each VM was provisioned with 2 CPU cores, 4GB RAM, and 100GB of storage, running on the Red Hat Enterprise Linux 8 (64-bit) operating system. To match the software environment of the original PNC paper <cit.>, we configured the system with Python 3.8.7, TensorFlow 2.8 and other matching packages.

The VM emulating the IoT device ran the encoder, while the VM emulating the edge server handled the decoder and frame reconstruction. For networking, we employed a custom TCP connection initiated by the IoT device using the Python sockets library. All frames were iteratively passed through this connection. Each frame was compressed and individually chunked into packets, which were then combined and decompressed at the edge server.

To facilitate the reception of smaller frames without socket blocking, a 3-byte delimiter was added for identification. Additionally, a 3-byte ACK delimiter was used as an acknowledgment signal, allowing the sender and receiver to coordinate when data transmission is permitted.

Network conditions were varied using the Linux Traffic Control Toolkit, a command-line tool that simulates network behavior such as delays, packet loss, and bandwidth limitations. A shell script modified the system to pre-set network configurations modeling different network qualities before initiating video transfer. 

Specifically, we tested with a wide range of network conditions. However, for clarity and simplicity in our experiments, we categorized the network conditions into three levels: minimal, medium, and high congestion. The main adjustable parameters in the Linux Traffic Control Toolkit are the data rate (limiting the maximum bandwidth available), burst size (defining the initial amount of data that can be sent at higher speeds before throttling to the set data rate), and latency (the time a packet is held in the buffer before getting processed or dropped). For high network congestion, the data rate was capped at 1 megabit per second, the burst size was set at 32 kilobits, and the latency was set to 400 ms. For medium congestion, the rate was set at 10 megabits per second, with a burst size of 64 kilobits and a latency of 200 ms. Finally, the arguments for minimal congestion were a rate of 50 megabits per second, a burst size of 128 kilobits, and latency of 50 ms.

The general dataflow, including measurements for AVC, PNC, and FrameCorr, remained consistent across all experiments with slight modifications to accommodate specific algorithm requirements. For instance, zero-padding was implemented for missing data depending on the reconstruction algorithm used (required by PNC but not FrameCorr). Our code then measured mean squared error, network latency, bandwidth, and other system metrics.

We opted for a virtual machine (VM) environment in our experiments to gain precise control over network conditions. This approach provided flexibility to simulate various real-world IoT scenarios without the cost and complexity of physical hardware and wireless equipment. Future work involves migrating our testbed to real IoT devices operating on a wireless channel.



 §.§ Reconstruction with Complete Data

We train both PNC and  on the dataset mentioned in Section <ref> with the objective of minimizing the reconstruction error. The encoder output dimension is set to 10. During training, we utilize the validation dataset to compute the loss after each epoch. We checkpoint the model with the lowest validation loss observed thus far. The training was run for 15 epochs.

The number of bytes a video contains serves as an indirect metric for measuring overall throughput, where more bytes take longer to send. 
This correlates with network bandwidth as higher bandwidth allows for higher throughput and vice versa. 

We present the number of bytes of encoded information for the 18 videos in our test set for PNC,  , and AVC in Figure <ref>. It is notable that AVC consistently requires fewer bytes to encode compared to PNC or .
Additionally, it is noteworthy that the number of bytes of encoded information for PNC remains the same as  across videos. This consistency is attributed to their fixed encoder output dimensions of 10.

The Mean Squared Error (MSE) is utilized as the metric to quantify the difference between a reconstructed frame and its original counterpart. Prior to passing through the encoding process, each pixel value is normalized to the range [0,1].

 

We compute the MSE of each video by first summing the squared differences of the pixel values between the original and reconstructed frames and then averaging those differences with respect to the number of frames embedded in each video. The MSE achieved by PNC, FrameCorr (no feature drops), and AVC are reported in Table <ref>. It turns out the MSE achieved by AVC is consistently lower than those achieved by PNC and , showing the success of traditional compression methods when no data is dropped.

Lastly, we recorded the total time it took for each method to process the entire dataset of videos or video frames as per Table <ref>. It's vital to note that both PNC and  took significantly longer (overall more than 3x greater than that of the highest AVC bitrate encoding whose CRF = 18) to process all the video frames. This is most likely due to the fact that 1) a sequence of extracted image frames consists of more bytes than the actual video themselves due to the innate compression techniques AVC embeds into the .mp4 videos and 2) the extra overhead associated with acknowledging packet transfer for not only every frame but also every feature. We also assumed that no timing constraint would be enforced, which also implies no feature vectors are dropped for PNC and . 






 §.§ Reconstruction with Partial Data


We quantified the percentage of video successfully transmitted for a single video clip under fluctuating network conditions and a set timing constraint (deadline), as documented in Table <ref>. Notably, transmission success was binary: either 0% or 100% of the video was successfully transferred for AVC, reflecting the integral encoding structure of .mp4 videos. Attempting to send a “partial" video would simply break its inherent structure and corrupt its data. In contrast, for PNC and FrameCorr, most if not all video frames were  transmitted. However, some features were omitted. Specifically, under minimal congestion, no features were dropped; under medium congestion, an average of 1-2 features per frame were omitted; and under high congestion, an average 3-4 features per frame were dropped.


AVC is unable to reconstruct videos with partial information since the codec requires complete data for the video to be properly decoded. As for PNC and , approximately 1-4 features (out of the 10 features) were dropped; we tasked the decoder with reconstructing the frame using the remaining features respectively and reported the MSE thereafter.

The MSE achieved by PNC and  for these scenarios is presented in Tables <ref> and <ref>. We report the results for  trained with K=1, which gives the best MSE values among values of K from 1 to 4. 



Unexpectedly, PNC surpasses  in nearly all video instances, indicating that simply zero-padding the absent segments performs admirably across most scenarios. Moreover, the MSE values rise with the escalation of dropped features, as anticipated. However, the marginal increase observed in both PNC and  demonstrates the resilience of these approaches when confronted with missing data events.










§ DISCUSSION

No loss of information. Our results suggest that if we can assume no loss of information, traditional video compression methods such as AVC demonstrate robust performance in reconstructing compressed data. However, training deep learning models poses its own set of hurdles. Firstly, generating a representative dataset may be impractical due to the diverse and intricate nature of real-world data sources. Secondly, model training necessitates fine-tuning hyperparameters and substantial computational resources.

If the application can accommodate data delivery delays, conventional ABR algorithms can dynamically adjust the bitrate in low-bandwidth scenarios to facilitate network data transmission (simply switch to the lower bitrate-encoded video for transfer). Consequently, we contend that in most situations, adopting a state-of-the-art traditional video compression algorithm remains the preferable approach.

Information Loss: Reconstructing frames with incomplete data poses a significant challenge for many video compression algorithms. Traditional methods like AVC require complete encoded information, and any partial loss can result in data corruption. In contrast, deep learning models can handle missing information through zero-padding, pixel prediction, etc. albeit with compromised reconstruction performance. Surprisingly, our study reveals that  underperforms compared to the state-of-the-art method, PNC. Several factors may account for this discrepancy:



  * Training Discrepancy: We train  to predict encoded information for a frame based on other encoded data of previous K frames. However, it's possible that the encoded information space differs significantly between the training, validation, and test sets. As a result,  may struggle to accurately predict the encoding for videos in the test set.
    
  * Model Complexity: Our experiments indicate that setting K=1 yields the best MSE value. This suggests that the simple two-layer neural network architecture used in  may not adequately capture the relationships among consecutive frames. Employing more sophisticated models like LSTMs could potentially improve performance in this context.


 Potential Workflows for 

Despite the limitations of , its deep learning-based frame reconstruction approach holds potential value for specific workflows. Consider a system severely constrained in network bandwidth (e.g., a consistently offline remote network) but possessing ample power and compute resources. In this scenario, sending highly condensed sets of features combined with reference frames, coupled with 's  reconstruction capabilities, could maximize the amount of usable data transmitted.  This approach leverages 's emphasis on local computation rather than network-intensive data transfer. While the high accessibility of the internet currently limits the prevalence of such use cases, there is value in exploring maximizing the potential of limited data through reconstruction, even beyond networking-oriented scenarios.



§ FUTURE WORK

Although  did not yield the most promising results, our extensive experimentation highlighted several opportunities to investigate alternative methodologies for improved outcomes and expand our testing into more realistic scenarios.

Integration of FrameCorr with Adaptive Bitrate Streaming. One potential exploration is the integration of the FrameCorr paradigm with ABR-based techniques. By combining FrameCorr's capability to reconstruct frames from partially received data with ABR's dynamic bitrate adjustment, there may be improvements in video quality and resilience to network fluctuations, particularly congested network conditions. 

Real-world Implementation on IoT Devices. Although our current experiments were conducted on machines hosted by a remote VM cluster, there is a need to validate our findings on actual IoT devices. Implementing and testing FrameCorr on physical IoT hardware, such as Raspberry Pi devices or other low-power embedded systems, can provide better insights into the practical challenges and performance impacts in real environments.

Furthermore, switching to live data capture or other similar workflow is another area of exploration that will not only scrutinize 's  flexibility in live, real-time streaming scenarios but also expose the system to greater I/O latency and storage limitations. 



§ CONCLUSION

Despite the prevalent challenge in video capture and processing systems of being unable to transmit complete data due to constraints such as limited time and bandwidth, traditional and deep-learning-based approaches appear to be somewhat ineffective in addressing this issue. Through experimentation with AVC, PNC, and ultimately our extension of PNC, , we found that AVC is unable to cope with partially received data. Conversely, PNC and  exhibit suboptimal performance.

Additionally, factors like power consumption, network variability, and hardware specifications demand highly specialized models and setups tailored to the specific use case. We hope this paper offers guidance on navigating the trade-offs for optimal model selection in other IoT/Edge-geared designs.

We are confident that  represents a notable step forward in addressing the challenge of effectively managing incomplete data. The current outcomes can be attributed to the selection of a basic model for , which we believe can be remedied through the adoption of a more suitable model and meticulous infrastructure. 


plain


