Linear-Gaussian Dual Control

QC 20261109

Abstract

This thesis investigates learning and decision making in linear-Gaussian dual control problems. A defining feature of dual control is that the applied input affects not only the current cost, but also the information available for future decisions and, consequently, the future cost. Input design therefore involves an inherent trade-off between exploitation, which seeks good immediate performance, and exploration, which improves future information.

We focus on a linear-Gaussian setting in which the physical state is removed and only the belief state of an unknown parameter is retained. Although simplified, this model preserves the essential information feedback that gives rise to the dual effect. It therefore provides a useful benchmark for isolating fundamental questions in dual control without the additional complications introduced by physical-state dynamics or more general uncertainty structures. Even in this setting, however, the resulting dynamic programming (DP) problem is nontrivial, and exact solutions over long horizons are generally difficult to obtain.

The thesis studies this problem from three complementary perspectives. First, we revisit the notions of dual effect and certainty-equivalence (CE) optimality at the level of a single Bellman recursion. We introduce a Bellman-side notion of no dual effect and characterize a nontrivial class of cost-to-go functions for which the certainty-equivalent input remains optimal, despite the input-dependent belief transition. Second, we develop a discounted infinite-horizon  framework for linear-Gaussian dual control. By introducing a weight tailored to the problem and working in the corresponding weighted sup-norm space, we establish contraction properties of the Bellman operators, characterize the optimal cost as a unique fixed point, and obtain convergence results for DP algorithms under unbounded belief and input spaces. Third, we investigate computationally tractable dual control policies. From a Bayesian perspective, short-horizon  solutions are used to construct receding- and cyclic-horizon approximations for longer horizons. From a frequentist perspective, we study perturbed CE policies and characterize structured exploration strategies through minimizing expected regret.

Together, these results provide structural, theoretical, and algorithmic insights into linear-Gaussian dual control, and establish a framework for studying how information acquisition affects sequential decision making.

Link to DiVA