Authorization

 

Journal

Volume 23, №3, Special Issue: Control, Teams, and Games

 
Controlling a Markov decision process with an abrupt change in the transition kernel (pp.325-345)
N. Dahlin1,⋆, S. Bose2, V.V. Veeravalli2
https://doi.org/10.30546/1683-6154.23.3.2024.325


1Department of Electrical and Computer Engineering, University at Albany, NY, USA,
2Department of Electrical and Computer Engineering and Coordinated Science Laboratory, University of Illinois Urbana-Champaign, IL USA,  e-mail: boses@illinois.edu,  vvv@illinois.edu
⋆Corresponding author e-mail:  
ndahlin@albany.edu

 

 

 

Abstract. We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.

 

Keywords: Piecewise Stationary Environment, Change Detection, Switched Markov Decision Process, Optimal Stopping, Sequence Approximations, Optimal Change, Detection Policy.

 

AMS Subject Classification: 90C40, 93E10, 60G.



COIA 2027

7-9 July 2027
Baku, Azerbaijan

Search