def policy_iteration(mdp):\n",
+ " """Solve an MDP by policy iteration [Figure 17.7]"""\n",
+ " U = {s: 0 for s in mdp.states}\n",
+ " pi = {s: random.choice(mdp.actions(s)) for s in mdp.states}\n",
+ " while True:\n",
+ " U = policy_evaluation(pi, U, mdp)\n",
+ " unchanged = True\n",
+ " for s in mdp.states:\n",
+ " a = argmax(mdp.actions(s), key=lambda a: expected_utility(a, s, U, mdp))\n",
+ " if a != pi[s]:\n",
+ " pi[s] = a\n",
+ " unchanged = False\n",
+ " if unchanged:\n",
+ " return pi\n",
+ "
Fortunately, it is not necessary to do _exact_ policy evaluation. \n",
+ "The utilities can instead be reasonably approximated by performing some number of simplified value iteration steps.\n",
+ "The simplified Bellman update equation for the process is\n",
+ "\n",
+ "$$U_{i+1}(s) \\leftarrow R(s) + \\gamma\\sum_{s'}P(s'\\ |\\ s,\\pi_i(s))U_{i}(s')$$\n",
+ "\n",
+ "and this is repeated _k_ times to produce the next utility estimate. This is called _modified policy iteration_."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ " Codestin Search App\n",
+ " \n",
+ " \n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "def policy_evaluation(pi, U, mdp, k=20):\n",
+ " """Return an updated utility mapping U from each state in the MDP to its\n",
+ " utility, using an approximation (modified policy iteration)."""\n",
+ " R, T, gamma = mdp.R, mdp.T, mdp.gamma\n",
+ " for i in range(k):\n",
+ " for s in mdp.states:\n",
+ " U[s] = R(s) + gamma * sum([p * U[s1] for (p, s1) in T(s, pi[s])])\n",
+ " return U\n",
+ "
This is the environment for our agent.\n",
+ "We assume for now that the environment is _fully observable_, so that the agent always knows where it is.\n",
+ "We also assume that the transitions are **Markovian**, that is, the probability of reaching state _s'_ from state _s_ only on _s_ and not on the history of earlier states.\n",
+ "Almost all stochastic decision problems can be reframed as a Markov Decision Process just by tweaking the definition of a _state_ for that particular problem.\n",
+ "
\n",
+ "However, the actions of our agent in this environment are unreliable.\n",
+ "In other words, the motion of our agent is stochastic. \n",
+ "More specifically, the agent does the intended action with a probability of _0.8_, but with probability _0.1_, it moves to the right and with probability _0.1_ it moves to the left of the intended direction.\n",
+ "The agent stays put if it bumps into a wall.\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "These properties of the agent are called the transition properties and are hardcoded into the GridMDP class as you can see below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ " Codestin Search App\n",
+ " \n",
+ " \n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ " def T(self, state, action):\n",
+ " if action is None:\n",
+ " return [(0.0, state)]\n",
+ " else:\n",
+ " return [(0.8, self.go(state, action)),\n",
+ " (0.1, self.go(state, turn_right(action))),\n",
+ " (0.1, self.go(state, turn_left(action)))]\n",
+ "
\n",
+ "Such a solution is known as a **policy** and is usually denoted by **π**.\n",
+ "
\n",
+ "The **optimal policy** is the policy that yields the highest expected utility an is usually denoted by **π* **.\n",
+ "
\n",
+ "The `GridMDP` class has a useful method `to_arrows` that outputs a grid showing the direction the agent should move, given a policy.\n",
+ "We will use this later to better understand the properties of the environment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ " Codestin Search App\n",
+ " \n",
+ " \n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ " def to_arrows(self, policy):\n",
+ " chars = {\n",
+ " (1, 0): '>', (0, 1): '^', (-1, 0): '<', (0, -1): 'v', None: '.'}\n",
+ " return self.to_grid({s: chars[a] for (s, a) in policy.items()})\n",
+ " def to_grid(self, mapping):\n",
+ " """Convert a mapping from (x, y) to v into a [[..., v, ...]] grid."""\n",
+ " return list(reversed([[mapping.get((x, y), None)\n",
+ " for x in range(self.cols)]\n",
+ " for y in range(self.rows)]))\n",
+ "
\n",
+ "\n",
+ "
\n",
+ "Notice that, because the cost of taking a step is fairly small compared with the penalty for ending up in `(4, 2)` by accident, the optimal policy is conservative. \n",
+ "In state `(3, 1)` it recommends taking the long way round, rather than taking the shorter way and risking getting a large negative reward of -1 in `(4, 2)`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Case 2\n",
+ "---\n",
+ "R(s) = -0.4 in all states except in terminal states"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "sequential_decision_environment = GridMDP([[-0.4, -0.4, -0.4, +1],\n",
+ " [-0.4, None, -0.4, -1],\n",
+ " [-0.4, -0.4, -0.4, -0.4]],\n",
+ " terminals=[(3, 2), (3, 1)])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> > > .\n",
+ "^ None ^ .\n",
+ "^ > ^ <\n"
+ ]
+ }
+ ],
+ "source": [
+ "pi = best_policy(sequential_decision_environment, value_iteration(sequential_decision_environment, .001))\n",
+ "from utils import print_table\n",
+ "print_table(sequential_decision_environment.to_arrows(pi))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This is exactly the output we expected\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "As the reward for each state is now more negative, life is certainly more unpleasant.\n",
+ "The agent takes the shortest route to the +1 state and is willing to risk falling into the -1 state by accident."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Case 3\n",
+ "---\n",
+ "R(s) = -4 in all states except terminal states"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "sequential_decision_environment = GridMDP([[-4, -4, -4, +1],\n",
+ " [-4, None, -4, -1],\n",
+ " [-4, -4, -4, -4]],\n",
+ " terminals=[(3, 2), (3, 1)])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> > > .\n",
+ "^ None > .\n",
+ "> > > ^\n"
+ ]
+ }
+ ],
+ "source": [
+ "pi = best_policy(sequential_decision_environment, value_iteration(sequential_decision_environment, .001))\n",
+ "from utils import print_table\n",
+ "print_table(sequential_decision_environment.to_arrows(pi))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This is exactly the output we expected\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The living reward for each state is now more negative than the most negative terminal. Life is so painful that the agent heads for the nearest exit as even the worst exit is less painful than the current state."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Case 4\n",
+ "---\n",
+ "R(s) = 4 in all states except terminal states"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "sequential_decision_environment = GridMDP([[4, 4, 4, +1],\n",
+ " [4, None, 4, -1],\n",
+ " [4, 4, 4, 4]],\n",
+ " terminals=[(3, 2), (3, 1)])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> > < .\n",
+ "> None < .\n",
+ "> > > v\n"
+ ]
+ }
+ ],
+ "source": [
+ "pi = best_policy(sequential_decision_environment, value_iteration(sequential_decision_environment, .001))\n",
+ "from utils import print_table\n",
+ "print_table(sequential_decision_environment.to_arrows(pi))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In this case, the output we expect is\n",
+ "\n",
+ "
\n",
+ "As life is positively enjoyable and the agent avoids _both_ exits.\n",
+ "Even though the output we get is not exactly what we want, it is definitely not wrong.\n",
+ "The scenario here requires the agent to anything but reach a terminal state, as this is the only way the agent can maximize its reward (total reward tends to infinity), and the program does just that.\n",
+ "
\n",
+ "Currently, the GridMDP class doesn't support an explicit marker for a \"do whatever you like\" action or a \"don't care\" condition.\n",
+ "You can however, extend the class to do so.\n",
+ "
\n",
+ "For in-depth knowledge about sequential decision problems, refer **Section 17.1** in the AIMA book."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "## Appendix\n",
+ "\n",
+ "Surprisingly, it turns out that there are six other optimal policies for various ranges of R(s). \n",
+ "You can try to find them out for yourself.\n",
+ "See **Exercise 17.5**.\n",
+ "To help you with this, we have a GridMDP editor in `grid_mdp.py` in the GUI folder. \n",
+ "
\n",
+ "Here's a brief tutorial about how to use it\n",
+ "
\n",
+ "Let us use it to solve `Case 2` above\n",
+ "1. Run `python gui/grid_mdp.py` from the master directory.\n",
+ "2. Enter the dimensions of the grid (3 x 4 in this case), and click on `'Build a GridMDP'`\n",
+ "3. Click on `Initialize` in the `Edit` menu.\n",
+ "4. Set the reward as -0.4 and click `Apply`. Exit the dialog. \n",
+ "\n",
+ "
\n",
+ "5. Select cell (1, 1) and check the `Wall` radio button. `Apply` and exit the dialog.\n",
+ "\n",
+ "
\n",
+ "6. Select cells (4, 1) and (4, 2) and check the `Terminal` radio button for both. Set the rewards appropriately and click on `Apply`. Exit the dialog. Your window should look something like this.\n",
+ "\n",
+ "
\n",
+ "7. You are all set up now. Click on `Build and Run` in the `Build` menu and watch the heatmap calculate the utility function.\n",
+ "\n",
+ "
\n",
+ "Green shades indicate positive utilities and brown shades indicate negative utilities. \n",
+ "The values of the utility function and arrow diagram will pop up in separate dialogs after the algorithm converges."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": []
}
],
"metadata": {
diff --git a/tests/test_mdp.py b/tests/test_mdp.py
index 9117a32d9..1aed4b58f 100644
--- a/tests/test_mdp.py
+++ b/tests/test_mdp.py
@@ -70,14 +70,6 @@ def test_policy_iteration():
(2, 1): (1, 0), (2, 2): (1, 0), (3, 0): (0, 1),
(3, 1): None, (3, 2): None}
- assert policy_iteration(sequential_decision_environment_3) == {
- (0, 0): (-1, 0), (0, 1): (0, -1), (0, 2): (0, -1), (0, 3): (0, -1), (0, 4): None,
- (1, 0): (-1, 0), (1, 1): (-1, 0), (1, 4): (1, 0),
- (2, 0): (-1, 0), (2, 1): (0, -1), (2, 2): None, (2, 4): (1, 0),
- (3, 0): (-1, 0), (3, 2): None, (3, 3): (1, 0), (3, 4): (1, 0),
- (4, 0): (-1, 0), (4, 3): (1, 0), (4, 4): (1, 0),
- (5, 0): None, (5, 1): (0, 1), (5, 2): (0, 1), (5, 3): (0, 1), (5, 4): (1, 0)}
-
def test_best_policy():
pi = best_policy(sequential_decision_environment,