Deep Q-Network - Reinforcement Learning with PyTorch

• Reinforcement Learning - Developing Intelligent Agents

• Deep Learning Course 5 of 6 - Level: Advanced

• Ref: DeepLizard – Reinforcement Learning - Developing Intelligent Agents

📚Project overview

We're going to be building and training a deep Q-network to learn to balance a pole on a moving cart. This is widely known as the cart and pole problem.

We'll be using OpenAI's Gym toolkit to set up our cart and pole environment.

Image Snapped from Deeplizard

Image Snapped from Packt – Hands-On Q-Learning with Python

---

🄠-Learning Algorithm

1. Initialize replay memory capacity.

2. Initialize the policy network with random weights.

3. Clone the policy network, and call it the target network.

4. For each episode:

• Initialize the starting state.

• For each time step:

  1. Select an action.
  • Via exploration or exploitation

  2. Execute selected action in an emulator.

  3. Observe reward and next state.

  4. Store experience in replay memory.

  5. Sample random batch from replay memory.

  6. Preprocess states from batch.

  7. Pass batch of preprocessed states to policy network.

  8. Calculate loss between output Q-values and target Q-values.

  • Requires a pass to the target network for the next state
  9. Gradient descent updates weights in the policy network to minimize loss.
  • After x time steps, weights in the target network are updated to the weights in the policy network.