When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multitimescale Resource Management for Multiaccess Edge Computing in 5G Ultradense Network

【Author】 Yu, Shuai; Chen, Xu; Zhou, Zhi; Gong, Xiaowen; Wu, Di

【Source】IEEE INTERNET OF THINGS JOURNAL

【Abstract】Recently, smart cities, healthcare system, and smart vehicles have raised challenges on the capability and connectivity of state-of-the-art Internet-of-Things (IoT) devices, especially for the devices in hotspots area. Multiaccess edge computing (MEC) can enhance the ability of emerging resource-intensive IoT applications and has attracted much attention. However, due to the time-varying network environments, as well as the heterogeneous resources of network devices, it is hard to achieve stable, reliable, and real-time interactions between edge devices and their serving edge servers, especially in the 5G ultradense network (UDN) scenarios. Ultradense edge computing (UDEC) has the potential to fill this gap, especially in the 5G era, but it still faces challenges in its current solutions, such as the lack of: 1) efficient utilization of multiple 5G resources (e.g., computation, communication, storage, and service resources); 2) low overhead offloading decision making and resource allocation strategies; and 3) privacy and security protection schemes. Thus, we first propose an intelligent UDEC (I-UDEC) framework, which integrates blockchain and artificial intelligence (AI) into 5G UDEC networks. Then, in order to achieve real-time and low overhead computation offloading decisions and resource allocation strategies, we design a novel two-timescale deep reinforcement learning (2Ts-DRL) approach, consisting of a fast-timescale and a slow-timescale learning process, respectively. The primary objective is to minimize the total offloading delay and network resource usage by jointly optimizing computation offloading, resource allocation, and service caching placement. We also leverage federated learning (FL) to train the 2Ts-DRL model in a distributed manner, aiming to protect the edge devices' data privacy. Simulation results corroborate the effectiveness of both the 2Ts-DRL and FL in the I-UDEC framework and prove that our proposed algorithm can reduce task execution time up to 31.87%.

【Keywords】Edge computing; Servers; Resource management; Cloud computing; 5G mobile communication; Machine learning; Blockchain; computation offloading; deep reinforcement learning (DRL); federated learning (FL); multiaccess edge computing (MEC); service caching; ultradense network (UDN)

【标题】当深度强化学习遇到联邦学习时：5G 超密集网络中多接入边缘计算的智能多时间尺度资源管理

【摘要】最近，智慧城市、医疗保健系统和智能车辆对最先进的物联网 (IoT) 设备的能力和连接性提出了挑战，尤其是热点区域的设备。多接入边缘计算（MEC）可以增强新兴资源密集型物联网应用的能力，备受关注。然而，由于网络环境的时变，以及网络设备资源的异构性，边缘设备与其服务的边缘服务器之间很难实现稳定、可靠、实时的交互，尤其是在5G超密集网络中(UDN) 场景。超密集边缘计算（UDEC）具有填补这一空白的潜力，尤其是在 5G 时代，但其目前的解决方案仍面临挑战，例如缺乏：1）多种 5G 资源（如计算、通信）的有效利用、存储和服务资源）； 2) 低开销卸载决策和资源分配策略； 3) 隐私和安全保护方案。因此，我们首先提出了一个智能UDEC（I-UDEC）框架，将区块链和人工智能（AI）集成到5G UDEC网络中。然后，为了实现实时和低开销的计算卸载决策和资源分配策略，我们设计了一种新颖的双时间尺度深度强化学习（2Ts-DRL）方法，包括一个快速时间尺度和一个慢时间尺度的学习过程， 分别。主要目标是通过联合优化计算卸载、资源分配和服务缓存放置来最小化总卸载延迟和网络资源使用。我们还利用联邦学习 (FL) 以分布式方式训练 2Ts-DRL 模型，旨在保护边缘设备的数据隐私。仿真结果证实了 2Ts-DRL 和 FL 在 I-UDEC 框架中的有效性，并证明我们提出的算法可以将任务执行时间减少高达 31.87%。

【关键词】边缘计算；服务器；资源管理;云计算; 5G移动通信；机器学习；区块链；计算卸载；深度强化学习（DRL）；联邦学习（FL）；多路访问边缘计算（MEC）；服务缓存；超密集网络 (UDN)

【发表时间】2021

【收录时间】2022-07-06

【文献类型】Article

【论文大主题】区块链联邦学习

【论文小主题】联邦学习为主体

【影响因子】10.238

【翻译者】石东瑛