Consider the discrete-time time-invariant one dimensional system with \(X=\mathbb{R}\), transitions $$ x^{+}=x+u, $$ and control value space the nonnegative reals: $$ \mathcal{U}=\mathbb{R}_{+} . $$ With \(q(t, x, u):=u^{2}, p(x):=x^{2}, \sigma=0\), and \(\tau=3\), find \(V(t, x)\) for all \(x\) and \(t=0,1,2\) using the dynamic programming technique. Next guess a general formula for arbitrary \(t, \sigma, \tau\) and establish its validity.

Suppose that \(L_{f} V(x) \leq 0\) for all \(x\) and that \(\dot{x}=f(x)+G(x) u\) is globally stabilized by \(u=-(\nabla V(x) \cdot G(x))^{\prime}\), as in Proposition 5.9.1. Show that \(u=k(x)\) is an optimal feedback, and \(V\) is the value function, for some suitably chosen cost. (Hint: Let $Q(x):=-L_{f} V(x)+\frac{1}{2} L_{G} V(x)\left(L_{G} V(x)\right)^{\prime}$, which gives (8.64) for which \(R\) ? Use Exercise 8.5.5.)

Find the value function \(V\) and the optimal feedback solution for the problem of minimizing \(\int_{0}^{\infty} x^{2}+u^{2} d t\) for the scalar system \(\dot{x}=x^{2}+u\). (Hint: The HJB equation is now an ordinary differential equation, in fact, a quadratic equation on the derivative of \(V\).)

Prove, without using the Pole-Shifting Theorem: If \(\Sigma\) is a controllable time-invariant linear discrete-time system over \(\mathbb{K}=\mathbb{R}\), then there exists an \(m \times n\) real matrix \(F\) such that \(A+B F\) is a convergent matrix.

Consider the case when \(B=0, Q=I\), and \(S\) approaches zero. Show that the formulas for least-squares observation in Section \(6.3\) can be recovered from the results in this section. (Hint: The equation for \(\widetilde{P}\) can be solved with final condition zero, and its solution at the initial time can be expressed directly in terms of the Wronskian \(W\).)

Consider the problem of minimizing the same cost (8.42) over the set of all trajectories of the system $$ \dot{\xi}=A \xi+G \bar{u}+B \omega, $$ where \(\bar{u}\) is a known control on \([\sigma, \tau]\). Show that the minimum is achieved through the solution of $$ \dot{z}=A z+L[C z-\bar{y}]+G \bar{u}, \quad z(\sigma)=0, $$ where \(L\) is the same as before. (Hint: Simply convert the original problem into an estimation problem for \(\dot{\widetilde{\xi}}=A \xi+B \omega\), where \(\tilde{\xi}:=\xi-\bar{x}\) and \(\bar{x}\) satisfies $\dot{\bar{x}}=A \bar{x}+G \bar{u} .)$