Heterogeneous Knowledge Transfer via Hierarchical Teaching in Cooperative Multiagent Reinforcement Learning

Heterogeneous knowledge naturally arises among different agents in cooperative multiagent reinforcement learning (MARL). Existing works have demonstrated that peer-to-peer knowledge transfer, a process referred to as teaching, accelerates individual learning speed and improves team-wide performance. Similar to recent learning to teach frameworks, we aim to learn teaching policies that decide when and what to advise to a teammate. In this work, we introduce a new learning to teach framework, called Hierarchical Multiagent Teaching (HMAT). The proposed framework solves difficulties faced by existing works when operating in domains with long horizons, large state spaces, and continuous actions. HMAT transfers heterogeneous knowledge by taking advantage of temporal abstractions of hierarchical reinforcement learning and representations of a deep neural network. Our empirical evaluations show that HMAT accelerates teamwide learning progress in complex environments.