ISR research accomplishments
Risk-Sensitive and Mixed Estimation and Control of Markov Decision Processes
Research team
Steven I. Marcus (ECE/ISR), S.P. Coraluppi, S. Dey, P. Fard, E. Fernández-Gaucherand, D. Hernández-Hernández
Risk-sensitive control
The motivation is risk sensitive control. Derived risk-sensitive and mixed
risk-neutral/minimax controllers and state estimators for discrete
time Markov processes. Proved results on the structure
of risk-sensitive controllers for Hidden Markov Models. Robust HMM Filtering
for Dynamic Classification and Prediction.
The problem was detecting or predicting a change in operating condition. The applications are condition-based maintenance in manufacturing (e.g. tool wear detection) and vehicle detection and classification in the battlefield. HMMs are over-confident and give unrealistic confidence scores for dynamic classification in the tool wear problem.
Robust HMM Filters are filters and state estimators (decoders) for Hidden Markov Models
(HMMs) that are robust with respect to noise models. Risk-sensitive filters (Dey,
Moore), based on risk-sensitive control (exponential sum of costs
– Baras, Marcus). Mixed estimators that minimize
an expected cost--with a constraint on the worst-case (Dey &
Marcus, 1998 IEEE Conf. on Decision and Control). Key point: Posterior probability distribution 
not a sufficient statistic
(information state)
Risk sensitive and mixed estimation
Observations
Similar to risk sensitive (r.s.) control,
but Ut replaced with state estimate 
Cost
Recursive equations for information state
and state estimate as in control case
Robustness of risk sensitive estimation. Penalizes higher-order moments
of m.s. estimation error. More robust (approaches minimax or
worst case) as 
. Is there a more direct way to
trade performance and robustness?
Mixed HMM Estimation
Minimize m.s. estimation error subject to a constraint on the worst-case estimation error. Can directly trade off performance and robustness. Mixed estimators that minimize an expected cost--with a constraint on the worst-case (Dey & Marcus, 1998 IEEE Conf. on Decision and Control). These estimators and controllers are more robust with respect to modeling errors. Built on work of Baras, James, & others. Invited address at 1996 Conference on Mathematical Theory of Networks and Systems. Applications in portfolio management (Stettner; Bielecki, Hernandez-Hernandez, and Pliska). Application to prediction of failure of drilling tools in the Center for Auditory and Acoustics Research.
For more information
S.P. Coraluppi & S.I. Marcus, "Mixed Risk-Neutral/Minimax Control of Discrete-Time, Finite-State Markov Decision Processes," IEEE Trans. Automatic Control, June 2000.
D. Hernández-Hernández, S.I. Marcus, & P. Fard, "Analysis of a Risk SensitiveControl Problem for Hidden Markov Chains,'' IEEE Trans. Automatic Control, 44, May 1999, 1093-1100.
S.I. Marcus, et. al., "Risk Sensitive Markov Decision Processes,'' in Systems and Control in theTwenty-First Century, C. I. Byrnes, et. al., eds. Boston: Birkhauser, 1997, 263-279.
S. Dey & S. I. Marcus, "A Framework for Mixed Estimation of Hidden Markov Models,'' Proc. 37th IEEE Conf. Decision and Control, Dec. 1998, Tampa, FL.
Additional references available at Steven I. Marcus' web site
