基本概念
State
$$
s_i\quad, \quad S = {s_i}
$$
- 表示状态和状态空间（集合）
Action
$$
a_i \quad , \quad A = {a_i}
$$
- 表示动作和动作空间（集合）
- 可用Tabular representation表示
Policy
$$
\pi \quad , \quad \pi (a_i | s_j) = c_{k}
$$
- 用概率形式表示动作可能的结果
- 针对一个状态的概率之和为1
- 可用Tabular representation表示
Deterministic policy （确定性情况）
对于一个状态S_j，一个动作a_i对他的概率为1，其余动作对该状态的概率均为0
Stochastic policy（不确定性情况）
不存在某一个动作对一个状态的概率为1
Reward
- positive reward -> encouragement
- negative reward -> punishment
$$
p(r=-1|s_1, a_1) = 1 \quad & \quad p(r \neq -1 | s_1,a_1) = 0
$$
Discount rate
$$
\gamma \in [0,1)
$$
Discounted return
$$
\begin{align}
\text{discounted return} &= p_1 + \gamma p_2 + \gamma ^2 p_3 + \gamma ^3 p_4 + \gamma ^4 p_5 + \gamma ^5 p_6 + \dots \
\text{In the case: }& p_1 =0 , p_2=0 , p_3=0 , p_4=1 , p_5=1 , p_6=1 \
\text{discounted return} &= \gamma ^3 (1+ \gamma + \gamma ^2 + \dots) \
&=\gamma ^3 \frac{1}{1-\gamma}.
\end{align}
$$
Roles:
-
the sum becomes finite;
-
balance the far and near future rewards:
-
$$
\text{If } \gamma \text{ is close to 0, the value of the discounted return is dominated by the rewards obtained in the near future.}
$$ -
$$
\text{If } \gamma \text{ is close to 1, the value of the discounted return is dominated by the rewards obtained in the far future.}
$$
-
Markov decision process (MDP)
Markov property: memoryless property （不具有记忆性：与历史无关）
$$
p(s_{t+1}|a_{t+1},s_t, \dots ,a_1,s_0) = p(s_{t+1}|a_{t+1},s_t), \
p(r_{t+1}|a_{t+1},s_t, \dots ,a_1,s_0) = p(r_{t+1}|a_{t+1},s_t).
$$
- Markov process 是带有概率的动作
- 被赋予了 policy 的 Markov process 是 Markov decision process