Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhanced mdp_apps notebook #782

Merged
merged 2 commits into from
Mar 1, 2018
Merged

Enhanced mdp_apps notebook #782

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
13e9070
Added pathfinding example
ad71 Feb 28, 2018
6848290
Added images
ad71 Feb 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added images/maze.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/mdp-d.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
193 changes: 166 additions & 27 deletions mdp_apps.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@
" - State dependent reward function\n",
" - State and action dependent reward function\n",
" - State, action and next state dependent reward function\n",
"\n",
"- Grid MDP\n",
" - Pathfinding problem\n",
"\n",
"## SIMPLE MDP\n",
"---\n",
Expand Down Expand Up @@ -221,7 +222,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"['study', 'pub', 'sleep', 'facebook', 'quit']\n"
"['quit', 'sleep', 'study', 'pub', 'facebook']\n"
]
}
],
Expand Down Expand Up @@ -294,7 +295,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"{'class3': 'pub', 'leisure': 'quit', 'class2': 'study', 'class1': 'study', 'end': None}\n"
"{'class2': 'sleep', 'class3': 'pub', 'end': None, 'class1': 'study', 'leisure': 'quit'}\n"
]
}
],
Expand All @@ -318,7 +319,7 @@
"data": {
"text/plain": [
"{'class1': 'study',\n",
" 'class2': 'study',\n",
" 'class2': 'sleep',\n",
" 'class3': 'pub',\n",
" 'end': None,\n",
" 'leisure': 'quit'}"
Expand Down Expand Up @@ -668,7 +669,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"['study', 'pub', 'sleep', 'facebook', 'quit']\n"
"['quit', 'sleep', 'study', 'pub', 'facebook']\n"
]
}
],
Expand Down Expand Up @@ -769,7 +770,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"{'class3': 'study', 'leisure': 'quit', 'class2': 'sleep', 'class1': 'facebook', 'end': None}\n"
"{'class2': 'sleep', 'class3': 'study', 'end': None, 'class1': 'facebook', 'leisure': 'quit'}\n"
]
}
],
Expand Down Expand Up @@ -832,46 +833,43 @@
"We have the following transition probability matrices:\n",
"<br>\n",
"<br>\n",
"Action 1: Cruising streets\n",
"<br>\n",
"Action 1: Cruising streets \n",
"<br>\n",
"$$\\\\\n",
"$\\\\\n",
" P^{1} = \n",
" \\left[ {\\begin{array}{ccc}\n",
" \\frac{1}{2} & \\frac{1}{4} & \\frac{1}{4} \\\\\n",
" \\frac{1}{2} & 0 & \\frac{1}{2} \\\\\n",
" \\frac{1}{4} & \\frac{1}{4} & \\frac{1}{2} \\\\\n",
" \\end{array}}\\right] \\\\\n",
" \\\\\n",
"$$\n",
" $\n",
"<br>\n",
"<br>\n",
"Action 2: Waiting at the taxi stand \n",
"Action 2: Waiting at the taxi stand \n",
"<br>\n",
"<br>\n",
"$$\\\\\n",
"$\\\\\n",
" P^{2} = \n",
" \\left[ {\\begin{array}{ccc}\n",
" \\frac{1}{16} & \\frac{3}{4} & \\frac{3}{16} \\\\\n",
" \\frac{1}{16} & \\frac{7}{8} & \\frac{1}{16} \\\\\n",
" \\frac{1}{8} & \\frac{3}{4} & \\frac{1}{8} \\\\\n",
" \\end{array}}\\right] \\\\\n",
" \\\\\n",
"$$\n",
" $\n",
"<br>\n",
"<br>\n",
"Action 3: Waiting for dispatch \n",
"<br>\n",
"<br>\n",
"$$\\\\\n",
"$\\\\\n",
" P^{3} =\n",
" \\left[ {\\begin{array}{ccc}\n",
" \\frac{1}{4} & \\frac{1}{8} & \\frac{5}{8} \\\\\n",
" 0 & 1 & 0 \\\\\n",
" \\frac{3}{4} & \\frac{1}{16} & \\frac{3}{16} \\\\\n",
" \\end{array}}\\right] \\\\\n",
" \\\\\n",
"$$\n",
" $\n",
"<br>\n",
"<br>\n",
"For the sake of readability, we will call the states A, B and C and the actions 'cruise', 'stand' and 'dispatch'.\n",
Expand Down Expand Up @@ -914,44 +912,41 @@
"<br>\n",
"Action 1: Cruising streets \n",
"<br>\n",
"<br>\n",
"$$\\\\\n",
"$\\\\\n",
" R^{1} = \n",
" \\left[ {\\begin{array}{ccc}\n",
" 10 & 4 & 8 \\\\\n",
" 14 & 0 & 18 \\\\\n",
" 10 & 2 & 8 \\\\\n",
" \\end{array}}\\right] \\\\\n",
" \\\\\n",
"$$\n",
" $\n",
"<br>\n",
"<br>\n",
"Action 2: Waiting at the taxi stand \n",
"<br>\n",
"<br>\n",
"$$\\\\\n",
"$\\\\\n",
" R^{2} = \n",
" \\left[ {\\begin{array}{ccc}\n",
" 8 & 2 & 4 \\\\\n",
" 8 & 16 & 8 \\\\\n",
" 6 & 4 & 2\\\\\n",
" \\end{array}}\\right] \\\\\n",
" \\\\\n",
"$$\n",
" $\n",
"<br>\n",
"<br>\n",
"Action 3: Waiting for dispatch \n",
"<br>\n",
"<br>\n",
"$$\\\\\n",
"$\\\\\n",
" R^{3} = \n",
" \\left[ {\\begin{array}{ccc}\n",
" 4 & 6 & 4 \\\\\n",
" 0 & 0 & 0 \\\\\n",
" 4 & 0 & 8\\\\\n",
" \\end{array}}\\right] \\\\\n",
" \\\\\n",
"$$\n",
" $\n",
"<br>\n",
"<br>\n",
"We now build the reward model as a dictionary using these matrices."
Expand Down Expand Up @@ -1194,7 +1189,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"['cruise', 'dispatch', 'stand']\n"
"['stand', 'dispatch', 'cruise']\n"
]
}
],
Expand Down Expand Up @@ -1290,6 +1285,150 @@
"We have successfully adapted the existing code to a different scenario yet again.\n",
"The takeaway from this section is that you can convert the vast majority of reinforcement learning problems into MDPs and solve for the best policy using simple yet efficient tools."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## GRID MDP\n",
"---\n",
"### Pathfinding Problem\n",
"Markov Decision Processes can be used to find the best path through a maze. Let us consider this simple maze.\n",
"![title](images/maze.png)\n",
"\n",
"This environment can be formulated as a GridMDP.\n",
"<br>\n",
"To make the grid matrix, we will consider the state-reward to be -0.1 for every state.\n",
"<br>\n",
"State (1, 1) will have a reward of -5 to signify that this state is to be prohibited.\n",
"<br>\n",
"State (9, 9) will have a reward of +5.\n",
"This will be the terminal state.\n",
"<br>\n",
"The matrix can be generated using the GridMDP editor or we can write it ourselves."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"grid = [\n",
" [None, None, None, None, None, None, None, None, None, None, None], \n",
" [None, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, None, +5.0, None], \n",
" [None, -0.1, None, None, None, None, None, None, None, -0.1, None], \n",
" [None, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, None], \n",
" [None, -0.1, None, None, None, None, None, None, None, None, None], \n",
" [None, -0.1, None, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, None], \n",
" [None, -0.1, None, None, None, None, None, -0.1, None, -0.1, None], \n",
" [None, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, None, -0.1, None], \n",
" [None, None, None, None, None, -0.1, None, -0.1, None, -0.1, None], \n",
" [None, -5.0, -0.1, -0.1, -0.1, -0.1, None, -0.1, None, -0.1, None], \n",
" [None, None, None, None, None, None, None, None, None, None, None]\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have only one terminal state, (9, 9)"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"terminals = [(9, 9)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We define our maze environment below"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"maze = GridMDP(grid, terminals)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To solve the maze, we can use the `best_policy` function along with `value_iteration`."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"pi = best_policy(maze, value_iteration(maze))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is the heatmap generated by the GridMDP editor using `value_iteration` on this environment\n",
"<br>\n",
"![title](images/mdp-d.png)\n",
"<br>\n",
"Let's print out the best policy"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None None None None None None None None None None None\n",
"None v < < < < < < None . None\n",
"None v None None None None None None None ^ None\n",
"None > > > > > > > > ^ None\n",
"None ^ None None None None None None None None None\n",
"None ^ None > > > > v < < None\n",
"None ^ None None None None None v None ^ None\n",
"None ^ < < < < < < None ^ None\n",
"None None None None None ^ None ^ None ^ None\n",
"None > > > > ^ None ^ None ^ None\n",
"None None None None None None None None None None None\n"
]
}
],
"source": [
"from utils import print_table\n",
"print_table(maze.to_arrows(pi))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can infer, we can find the path to the terminal state starting from any given state using this policy.\n",
"All maze problems can be solved by formulating it as a MDP."
]
}
],
"metadata": {
Expand Down