RL1 - 基本概念 academic 强化学习 约 841 字 大约 3 分钟 主要框架 State(状态) ：The status of the agent with respect to the environment.
State Space(状态空间) : 所有状态的集合。S = { s i } i = 1 n S=\{s_i\}_{i=1}^{n} S = { s i } i = 1 n
Action(动作) : 对于每一个状态，都有可选择的动作。
Action space of a state : 对应状态中所有可选择的动作集合。A ( s i ) = { a i } i = 1 n A(s_i)=\{a_i\}_{i=1}^{n} A ( s i ) = { a i } i = 1 n
State transition(状态转换) : s 1 → a 1 s 2 s_1\overset{a_1}{\rightarrow} s_2 s 1 → a 1 s 2
State transition probability : p ( s 2 ∣ s 1 , a 1 ) p(s_2|s_1,a_1) p ( s 2 ∣ s 1 , a 1 ) s 1 s_1 s 1 a 1 a_1 a 1 s 2 s_2 s 2
Policy π \pi π : 指导agent在当前状态下选择哪个动作。
Reward(奖励) : 在执行一个动作后获得的一个常数(依赖于当前状态和所采取的动作)。同样可以用条件概率的形式进行描述，如p ( r = 1 ∣ s 1 , a 1 ) p(r=1|s_1,a_1) p ( r = 1∣ s 1 , a 1 ) s 1 s_1 s 1 a 1 a_1 a 1 r = 1 r=1 r = 1
Trajectory ：a state-action-reward chain.(可以有限，也可以是无限长的trajectory) s 1 → r = 0 a 2 s 2 → r = 0 a 2 s 5 → r = 0 a 2 s 8 → r = 1 a 2 s 9 s_1\overset{a_2}{\underset{r=0}{\rightarrow}}s_2\overset{a_2}{\underset{r=0}{\rightarrow}}s_5\overset{a_2}{\underset{r=0}{\rightarrow}}s_8\overset{a_2}{\underset{r=1}{\rightarrow}}s_9 s 1 r = 0 → a 2 s 2 r = 0 → a 2 s 5 r = 0 → a 2 s 8 r = 1 → a 2 s 9
Return of a trajectory ：将对应的轨迹所获得的所有reward的总和，可以粗步衡量一个策略的好坏。
Discounted return(of a trajectory) ：为了应对具有无限步的trajectory的r e t u r n = ∞ return=\infty re t u r n = ∞ s 1 → r = 0 a 2 s 2 → r = 0 a 2 s 5 → r = 0 a 2 s 8 → r = 1 a 2 s 9 → r = 1 a 2 s 9 → r = 1 a 2 s 9 … s_1\overset{a_2}{\underset{r=0}{\rightarrow}}s_2\overset{a_2}{\underset{r=0}{\rightarrow}}s_5\overset{a_2}{\underset{r=0}{\rightarrow}}s_8\overset{a_2}{\underset{r=1}{\rightarrow}}s_9\color{blue}{\overset{a_2}{\underset{r=1}{\rightarrow}}s_9\overset{a_2}{\underset{r=1}{\rightarrow}}s_9\dots} s 1 r = 0 → a 2 s 2 r = 0 → a 2 s 5 r = 0 → a 2 s 8 r = 1 → a 2 s 9 r = 1 → a 2 s 9 r = 1 → a 2 s 9 … r e t u r n = 0 + 0 + 0 + 1 + 1 + ⋯ = ∞ return=0+0+0+1+1+\dots=\infty re t u r n = 0 + 0 + 0 + 1 + 1 + ⋯ = ∞ discount rate , γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) d i s c o u n t e d r a t e = 0 + γ 0 + γ 2 0 + γ 3 1 + γ 4 1 + ⋯ = γ 3 1 1 − γ discounted\space rate=0+\gamma 0+\gamma^2 0+\gamma^3 1+\gamma^4 1+\dots=\gamma^3 \frac{1}{1-\gamma} d i sco u n t e d r a t e = 0 + γ 0 + γ 2 0 + γ 3 1 + γ 4 1 + ⋯ = γ 3 1 − γ 1 γ \gamma γ γ \gamma γ
Episode(trial) ：When interacting with the environment following a policy, the agent may stop at some terminal states . The resulting trajectory is called an episode(or a trial)/ 即表示具有终止状态terminal states 的trajectory，通常是具有有限步长的trajectory. 同理，这样的任务称为episodic tasks 。
continuing tasks ：即不具备terminal states的任务，会与环境一直交互下去。 可以通过设置将episodic tasks转换成continuing tasks，如可以在target states中限制action space，控制其一直待在target states中。 Deterministic — Stochastic
关键元素：
Sets： State：the set of states S \mathcal{S} S Action：the set of actions A ( s ) \mathcal{A}(s) A ( s ) s ∈ S s\in\mathcal{S} s ∈ S Reward：the set of rewards R ( s , a ) \mathcal{R}(s,a) R ( s , a ) Probability distribution： State transition probability p ( s ′ ∣ s , a ) \color{blue}{p(s'|s,a)} p ( s ′ ∣ s , a ) s ′ s' s ′ Reward probability p ( r ∣ s , a ) \color{blue}{p(r|s,a)} p ( r ∣ s , a ) r r r Policy：at state s \mathcal{s} s a \mathcal{a} a π ( a ∣ s ) \pi(a|s) π ( a ∣ s ) Markov property：即无记忆的特性。 p ( s t + 1 ∣ a t + 1 , s t , … , a 1 , s 0 ) = p ( s t + 1 ∣ a t + 1 , s t ) p(s_{t+1}|a_{t+1},s_t,\dots,a_1,s_0)=p(s_{t+1}|a_{t+1},s_t) p ( s t + 1 ∣ a t + 1 , s t , … , a 1 , s 0 ) = p ( s t + 1 ∣ a t + 1 , s t ) r ( s t + 1 ∣ a t + 1 , s t , … , a 1 , s 0 ) = p ( r t + 1 ∣ a t + 1 , s t ) r(s_{t+1}|a_{t+1},s_t,\dots,a_1,s_0)=p(r_{t+1}|a_{t+1},s_t) r ( s t + 1 ∣ a t + 1 , s t , … , a 1 , s 0 ) = p ( r t + 1 ∣ a t + 1 , s t ) Markov process ：在policy是确定的情况下，MDP就变为MP。
下一页
RL2 - 贝尔曼公式
