Enhancing Information Freshness: An AoI Optimized Markov Decision Process Dedicated In the Underwater Task
Jingzehua Xu Yimian Ding Yiyuan Yang Shuai Zhang










Enhancing Information Freshness: An AoI Optimized Markov Decision Process Dedicated In the Underwater Task
    

Jingzehua Xu1^,+,
Yimian Ding1^,+,
Yiyuan Yang2,
Shuai Zhang3

1Tsinghua Shenzhen International Graduate School, Tsinghua University, China
2Department of Computer Science, University of Oxford, United Kingdom
3Department of Data Science, New Jersey Institute of Technology, USA
Email: {xjzh23, dingym24}@mails.tsinghua.edu.cn, yiyuan.yang@cs.ox.ac.uk, sz457@njit.edu

^+ These authors contributed equally to this work.

     
==========================================================================================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
Ocean exploration utilizing autonomous underwater vehicles (AUVs) via reinforcement learning (RL) has emerged as a significant research focus. However, underwater tasks have mostly failed due to the observation delay caused by acoustic communication in the Internet of underwater things. In this study, we present an AoI optimized Markov decision process (AoI-MDP) to improve the performance of underwater tasks. Specifically, AoI-MDP models observation delay as signal delay through statistical signal processing, and includes this delay as a new component in the state space. Additionally, we introduce wait time in the action space, and integrate AoI with reward functions to achieve joint optimization of information freshness and decision-making for AUVs leveraging RL for training. Finally, we apply this approach to the multi-AUV data collection task scenario as an example. Simulation results highlight the feasibility of AoI-MDP, which effectively minimizes AoI while showcasing superior performance in the task. To accelerate relevant research in this field, the code for simulation will be released as open-source in the future.



Age of Information, Markov Decision Process, Statistical Signal Processing, Reinforcement Learning, Autonomous Underwater Vehicles




§ INTRODUCTION

Harsh ocean environment put forward higher difficulty on ocean exploration <cit.>. As a novel approach, utilizing autonomous underwater vehicles (AUVs) via reinforcement learning (RL) has merged as a significant research focus <cit.>. Relying on Internet of underwater things (IoUT) <cit.>, AUVs can communicate with each other and work in collaboration to accomplish human-insurmountable tasks <cit.>. However, underwater tasks have mostly failed due to the observation delay caused by acoustic communication, leading to the non-causality of control policies <cit.>. Although this issue can be alleviated by introducing states that incorporate past information and account for the future effects of control laws <cit.>, it becomes increasingly challenging as the number of AUVs grows, leading to more complexity in both communication and decision-making processes <cit.>.

As a significant indicator evaluation the freshness of information, age of information (AoI) is proposed to measure the time elapsed at the receiver since the last information was generated until the most recent information is received <cit.>. And it has been verified to solve the severe delay caused by constantly sampling and transmitting observation information <cit.>. Central to this consensus is that minimizing AoI can enhance the freshness of information, thereby facilitating the efficiency of subsequent decision-making process in the presence of observation delay <cit.>. Currently, numerous studies have focused on optimizing AoI to aid decision-making in the context of land-based or underwater tasks. For example, Messaoudi et al. optimized vehicle trajectories relying on minimizing average AoI while reducing energy consumption <cit.>. Similarly, Lyu et al. leveraged AoI to assess transmission delay impacts on state estimation, improving performance under energy constraints <cit.>. These studies primarily aim to reduce AoI by improving the motion strategies of agents, without considering the impact of information update strategies on AoI. They assume that agents instantaneously perform the current action upon receiving the previous information. However, this zero-wait strategy has been shown to be suboptimal in scenarios with high variability in delay times <cit.>. Conversely, it has been demonstrated that introducing a waiting period before updating can achieve lower average AoI. This highlights the necessity of integrating optimized information update strategies into the underwater tasks. 

Furthermore, most studies currently leverage the standard Markov decision process (MDP) without observation delay to model the underwater tasks, which assumes the AUV can instantaneously receive current state information without delay, so that it can make corresponding actions <cit.>. This idealization, however, may not hold in many practical scenarios, since signal propagation delays and high update frequencies causing channel congestion reduce the freshness of received information, hindering the AUV’s decision-making efficiency. Therefore, extending the standard MDP framework to incorporate observation delays and AoI is necessary <cit.>.

Based on the above analysis, we attempted to propose an AoI optimized MDP (AoI-MDP) dedicated in the underwater task to improve the performance of the tasks with observation delay. The contributions of this paper include the following:


  * To the best of our knowledge, we are the first to formulate the underwater task as an MDP that incorporates observation delay and AoI. Based on AoI-MDP, we utilize RL for AUV training to realize joint optimization of both information updating and decision-making strategies.


  * Instead of simply modeling observation delay as a random distribution or stationary stochastic variable, we utilize statistical signal processing to realize the high-precision modeling via AUV equipped sonar, which potentially yielding more realistic results.


  * Through comprehensive evaluations and ablation experiments in the underwater data collection task, our AoI-MDP showcases superior feasibility and excellent performance in balancing multi-objective optimization. And to accelerate relevant research in this field, the code for simulation will be released as open-source in the future.





§ METHODOLOGY

In this section, we present the proposed AoI-MDP, which consists of three main components: an observation delay-aware state space, an action space that incorporates wait time, and reward functions based on AoI. To achieve high-precision modeling, AoI-MDP utilizes statistical signal processing (SSP) to represent observation delay as underwater acoustic signal delay, thereby aiming to minimize the gap between simulation and real-world underwater tasks.


 §.§ AoI Optimized Markov Decision Process

As illustrated in Fig. 1, consider the scenario where the i-th underwater acoustic signal is transmitted from the AUV at time T_i, and the corresponding observed information is received at time D_i, AoI is defined using a sawtooth piecewise function

    Δ(t)=t-T_i,D_i≤ t<D_i+1, ∀ i ∈𝐍.


Hence, we denote the MDP that integrates observation delay and characterizes the freshness of information through the AoI as the AoI-MDP, which can be defined by a quintuple Ω for further RL training <cit.> 




    Ω={𝒮, 𝒜, ℛ,  Pr(s_i+1|s_i, a_i), γ} ,

where 𝒮, 𝒜, ℛ represent state space, action space and reward functions, respectively. The term Pr(s_i+1|s_i, a_i) ∈ [0, 1] indicates state transition probability distribution, while γ ∈ [0, 1] represents a discount factor.

In AoI-MDP, instead of simply incorporating AoI as a component of the reward functions to guide objective optimization through RL training, we also leverage AoI as crucial side information to facilitate decision making. Specifically, we reformulate the standard MDP's state space, action space, and reward functions. The detailed designs for each of these elements are as follows:

State Space 𝒮: the state space of the AoI-MDP consists of two parts:  AUV's observed information s^'_i, and observation delay Y_i at time i, represented by s_i=(s^'_i,Y_i)∈𝒮^'×Y.We introduce the observation delay Y_i as a new element so that the AUV can be aware of the underwater acoustic signal delay when the sonar emits an underwater acoustic signal to detect the surrounding environment. Additionally, we achieve high-precision modeling of both s^'_i and Y_i through SSP, whose details are presented in Section 2-B. 

Action Space 𝒜: the action space of the AoI-MDP consists of the tuple a_i=(a_i^', Z_i)∈𝒜^'×Z, where a_i denotes the actions taken by the AUV, while Z_i indicates the wait time between observing the environmental information and decision-making at time i. Through jointly optimizing the wait time Z_i and action a_i, we aim to minimize the AoI, enabling the AUV's decision-making policy to converge to an optimal level.

Reward Function ℛ: the reward function r_i^' in standard MDP comprises elements with different roles, such as penalizing failures, promoting efficiency, and encouraging cooperation, etc. Here, we introduce the time-averaged AoI as a new component of the reward function. Thus, the updated reward function can be represented by the tuple r_i=(r^'_i, -Δ). And the time averaged AoI can be computed as follows:

    Δ=∑_i=2^𝒩((2Y_i-1+Y_i+Z_i)×(Y_i+Z_i))/2×(∑_i=1^𝒩Z_i+∑_i=1^𝒩Y_i),

where 𝒩 is the length of information signal. Therefore, the time averaged AoI can be minimized through RL training.

According to the above analysis, the total reward function is set below:

    R_i=∑_k=1^∞λ^(k)r^(k)_i,

where λ^(k) represents the weighting coefficient of the k-th reward function.

Based on proposed AoI-MDP, we further integrate it with RL training for the joint optimization of
information freshness and decision-making for AUVs. The pseudocode for the AoI-MDP based RL training is showcased in Algorithm 1.






 §.§ Observation Delay and Information Modeling

Different from previous work, our study enhances the state space of AoI-MDP by considering the observed information using estimated information perceived by AUV-equipped sensors. And we consider observation delay as underwater acoustic signal delay, rather than merely treating it as a random distribution <cit.> or stationary stochastic variable <cit.>. This approach aims to provide high-precision modeling to improve the performance in the underwater environment. And the schematic diagram is shown in Fig. 2.




To be specific, our study assumes the AUV leverages a sonar system to estimate the distance from itself to environmental objects. This was achieved by transmitting acoustic signals through sonar, measuring the time delay taken for these signals to propagate to the target, reflect, and return to the hydrophone, thus allowing for distance estimation. The acoustic signal propagation can be represented as

    𝒳[n]=𝒮[n-Y_i]+𝒲[n], n=0,1,…,N-1,

where 𝒮[n] represents the known signal, while Y_i denotes the time delay to be estimated, and 𝒲[n] is the Gaussian white noise with variance σ^2.

We further employ the flow correlator as an estimator to determine the time delay. Specifically, this estimator carries out the following computations on each received signal:


    J[Y_i]=∑_n=Y_i^Y_i+M-1   𝒳[n]𝒮[n-Y_i], 0≤ Y_i≤ N-M,
       Ŷ_̂î =  argmax[J[Y_i] ],


where M is the sampling length of 𝒮[n]. By calculating the value of Ŷ_̂î that maximizes the value of Eq. (6a), the estimated time delay value can be obtained through Eq. (6b).


On the other hand, the AUV in our study utilizes a long linear array sensor to estimate the azimuth β between its orientation and environmental objects. The signal propagation can be expressed as follows:


    x [n] =     A c o s [2 π(F_0d/c c o s β) n + ϕ] + 𝒲[n] , 
        n = 0,1 , ··· , M - 1,

where F_0 denotes the frequency of transmitted signal, while d represents the interval between sensors. Besides, c indicates the speed of underwater acoustic signal propagation, while A and ϕ are unknown signal amplitude and phase, respectively.

The estimator in SSP is further leveraged to estimate the azimuth β. By maximizing the spatial period graph, the estimate of β  (0βπ/2) can be calculated



    I_s(β) =1/M   (|. ∑_n = 0^M - 1 x [n]  exp[. - j 2 π(. F_0d/c c o s β.)n ]. |.)^2,
               β̂ =  argmax[I_s(β)].



By calculating the value of β that maximizes the value of Eq. (8a), the estimated time delay value can be obtained through Eq. (8b).

Finally, the AUV can achieve target positioning using the observed Ŷ_̂î and β̂. These observations are then utilized as data for the observed information in the state space of the AoI-MDP, which potentially
yields more realistic results to improve the underwater performance, while reducing the gap between simulation and reality in the underwater tasks.



§ EXPERIMENTS

In this section, we validate the proposed AoI-MDP through extensive simulation experiments. Further, we present the experimental results with further analysis and discussion.


 §.§ Task Description and Settings

Since open-sour underwater tasks are scarce, we consider the scenario of a multi-AUV data collection task as a classic example to evaluate the feasibility and effectiveness of the AoI-MDP. This task utilizes RL algorithms to train AUVs to collect data of sensor nodes in the Internet of underwater things, encompassing multiple objectives, such as maximizing sum data rate and collision avoidance, while minimizing energy consumption, etc. For the remaining details and parameters of the task, please refer to the previous work <cit.>.



 §.§ Experiment Results and Analysis

We first compared the experimental results of RL training based on AoI-MDP and standard MDP under identical conditions, respectively. Results in Fig. 3 show that AoI-MDP results in lower time-averaged AoI, reduced energy consumption, higher sum data rate, and greater cumulative rewards. This demonstrates that AoI-MDP improves the training effectiveness and performance of the RL algorithm.


















































































































































Then we evaluated the generalization performance of AoI-MDP using commonly employed delay models in the communication field, including exponential, poisson and geometric distributions. The experimental results, compared with the SSP model, are shown in Table 1. The AoI-MDP based RL training demonstrates superior performance across various distributions, indicating strong generalization capabilities. Additionally, SSP for time delay modeling achieved near-optimal results in AoI optimization, sum data rate optimization, and energy consumption optimization, underscoring its effectiveness in the underwater data collection task.

We further turned our attention to comparing the generalization of AoI-MDP in various RL algorithms. We conducted experiments utilizing AoI-MDP on soft actor-critic (SAC) and conservative Q-learning (CQL), within the contexts of online and offline RL, respectively. As shown in Fig. 4, both online and offline RL algorithms can successfully adapt to AoI-MDP, while ultimately achieving favorable training outcomes.









Finally we guided the multi-AUV in the underwater data collection task using the expert policy trained via SAC algorithm based on AoI-MDP and standard MDP respectively. As illustrated in Fig. 5, the trajectory coverage trained under AoI-MDP is more extensive, leading to more effective completion of the data collection task. Conversely, under standard MDP, the trajectories of AUVs appears more erratic, with lower node coverage, thereby showcasing suboptimal performance.



§ CONCLUSION

In this study, we propose AoI-MDP to improve the performance of underwater tasks. AoI-MDP models observation delay as signal delay through SSP, and includes this delay as a new component in the state space. Additionally, AoI-MDP introduces wait time in the action space, and integrate AoI with reward functions to achieve joint optimization of information freshness and decision making for AUVs leveraging RL for training. Simulation results highlight the feasibility, effectiveness and generalization of AoI-MDP over standard MDP, which effectively minimizes AoI while showcasing superior performance in the underwater task. The simulation code will be released as open-source to advance research in the future.

IEEEtran


10

url@samestyle








1
Z. Wang, Z. Zhang, J. Wang, C. Jiang, W. Wei, and Y. Ren, “Auv-assisted node repair for iout relying on multiagent reinforcement learning,” IEEE Internet of Things Journal, vol. 11, no. 3, pp. 4139–4151, 2024.

2
Y. Li, L. Liu, W. Yu, Y. Wang, and X. Guan, “Noncooperative mobile target tracking using multiple auvs in anchor-free environments,” IEEE Internet of Things Journal, vol. 7, no. 10, pp. 9819–9833, 2020.

8
R. H. Jhaveri, K. M. Rabie, Q. Xin, M. Chafii, T. A. Tran, and B. M. ElHalawany, “Guest editorial: Emerging trends and challenges in internet-of-underwater-things,” IEEE Internet of Things Magazine, vol. 5, no. 4, pp. 8–9, 2022.

3
Z. Zhang, J. Xu, G. Xie, J. Wang, Z. Han, and Y. Ren, “Environment and energy-aware auv-assisted data collection for the internet of underwater things,” IEEE Internet of Things Journal, vol. 11, no. 15, pp. 26 406–26 418, 2024.

4
W. Wei, J. Wang, J. Du, Z. Fang, C. Jiang, and Y. Ren, “Underwater differential game: Finite-time target hunting task with communication delay,” in ICC 2022 - IEEE International Conference on Communications, 2022, pp. 3989–3994.

5
J. Wu, C. Song, J. Ma, J. Wu, and G. Han, “Reinforcement learning and particle swarm optimization supporting real-time rescue assignments for multiple autonomous underwater vehicles,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 6807–6820, 2022.

6
R. Talak, S. Karaman, and E. Modiano, “Optimizing information freshness in wireless networks under general interference constraints,” IEEE/ACM Transactions on Networking, vol. 28, no. 1, pp. 15–28, 2020.

7
R. D. Yates, Y. Sun, D. R. Brown, S. K. Kaul, E. Modiano, and S. Ulukus, “Age of information: An introduction and survey,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 5, pp. 1183–1210, 2021.

10279435
K. Messaoudi, O. S. Oubbati, A. Rachedi, and T. Bendouma, “Uav-ugv-based system for aoi minimization in iot networks,” in ICC 2023 - IEEE International Conference on Communications, 2023, pp. 4743–4748.

9442098
L. Lyu, Y. Dai, N. Cheng, S. Zhu, Z. Ding, and X. Guan, “Cooperative transmission for aoi-penalty aware state estimation in marine iot systems,” in 2020 IEEE 18th International Conference on Industrial Informatics (INDIN), vol. 1, 2020, pp. 865–869.

8000687
Y. Sun, E. Uysal-Biyikoglu, R. D. Yates, C. E. Koksal, and N. B. Shroff, “Update or wait: How to keep your data fresh,” IEEE Transactions on Information Theory, vol. 63, no. 11, pp. 7492–7508, 2017.

Howard1960DynamicPA
R. A. Howard, “Dynamic programming and markov processes,” 1960. [Online]. Available: <https://api.semanticscholar.org/CorpusID:62124406>


10.1145/149439.133106
E. Altman and P. Nain, “Closed-loop control with delayed information,” SIGMETRICS Perform. Eval. Rev., vol. 20, no. 1, p. 193–204, jun 1992.

9
B. Jiang, J. Du, C. Jiang, Z. Han, and M. Debbah, “Underwater searching and multiround data collection via auv swarms: An energy-efficient aoi-aware mappo approach,” IEEE Internet of Things Journal, vol. 11, no. 7, pp. 12 768–12 782, 2024.

10
E. Altman and P. Nain, “Closed-loop control with delayed information,” in Proceedings of the 1992 ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems, ser. SIGMETRICS '92/PERFORMANCE '92.1em plus 0.5em minus 0.4emNew York, NY, USA: Association for Computing Machinery, 1992, p. 193–204.

11
K. Katsikopoulos and S. Engelbrecht, “Markov decision processes with delays and asynchronous cost collection,” IEEE Transactions on Automatic Control, vol. 48, no. 4, pp. 568–574, 2003.





