JournalVolume 23, №3, Special Issue: Control, Teams, and GamesControlling a Markov decision process with an abrupt change in the transition kernel (pp.325-345)
N. Dahlin1,⋆, S. Bose2, V.V. Veeravalli2 https://doi.org/10.30546/1683-6154.23.3.2024.3251Department of Electrical and Computer Engineering, University at Albany, NY, USA, 2Department of Electrical and Computer Engineering and Coordinated Science Laboratory, University of Illinois Urbana-Champaign, IL USA, ⋆Corresponding author e-mail: ndahlin@albany.edu
Abstract. We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy. Keywords: Piecewise Stationary Environment, Change Detection, Switched Markov Decision Process, Optimal Stopping, Sequence Approximations, Optimal Change, Detection Policy. AMS Subject Classification: 90C40, 93E10, 60G. |
|