To install this projects dependencies, run
pip install -r requirements.txt
Our project is dependent on OpenAI's Multi-agent Particle Environment with installation instructions here.
The environment used is out of date, so in order to run it, the changes detailed here must be made in the library.
The required modifications are in the file \maddpg-formation-control\src\multiagent\multiagent\multi_discrete.py"
Comment out gym.spaces import prng in line 7,
and change line 33 from
random_array = prng.np_random.rand(self.num_discrete_space)
to
random_array = np.random.RandomState().rand(self.num_discrete_space)
If you encounter the following error
ImportError: cannot import name 'reraise' from 'gym.utils'
Open maddpg-formation-control\src\multiagent\multiagent\rendering.py and comment out from gym.utils import reraise in line 14,
change line 20 from
reraise(suffix="HINT: you can install pyglet directly via 'pip install pyglet'. But if you really just want to install all Gym dependencies and not have to think about it, 'pip install -e .[all]' or 'pip install gym[all]' will do it.")
to
raise ImportError("HINT: you can install pyglet directly via 'pip install pyglet'. But if you really just want to install all Gym dependencies and not have to think about it, 'pip install -e .[all]' or 'pip install gym[all]' will do it.")
and line 25 from
reraise(prefix="Error occured while running `from pyglet.gl import *`",suffix="HINT: make sure you have OpenGL install. On Ubuntu, you can run 'apt-get install python-opengl'. If you're running on a server, you may need a virtual frame buffer; something like this should work: 'xvfb-run -s \"-screen 0 1400x900x24\" python <your_script.py>'")
to
raise ImportError("Error occured while running `from pyglet.gl import *")
It is also likely that you will encounter the following error
AttributeError: 'ImageData' object has no attribute 'data'
if you do, change the file \maddpg-formation-control\src\multiagent\multiagent\rendering.py" in line 101 from
arr = np.fromstring(image_data.data, dtype=np.uint8, sep='')
to
arr = np.fromstring(image_data.get_data(), dtype=np.uint8, sep='')
We define two different training scripts depending on whether we want to train a centralized or decentralized agent.
To train a centralized DDPG agent for 1000 episodes in a formation control with collision avoidance scenario, you can run the following command
python centralized_experiment.py
Further control over the experiments can be gained through use of the command line options, defined as
usage: centralized_experiment.py [-h] [--scenario SCENARIO_NAME] [--num_eps NUM_EPS] [--save_images IMAGES] [--save_models SAVE_MODEL] [--load_models LOAD_MODEL] [--train TRAIN]
[--save_suffix SAVE_SUFFIX] [--load_suffix LOAD_SUFFIX]
File to run experiments for som scenario with a centralized agent.
optional arguments:
-h, --help show this help message and exit
--scenario SCENARIO_NAME
Name of the scenario we want to run: formation_w_coll_avoidance, formation_w_goal or simple_formation
--num_eps NUM_EPS Number of episodes to train for.
--save_images IMAGES True to save images and gifs, anything else not to.
--save_models SAVE_MODEL
True to save models, anything else not to.
--load_models LOAD_MODEL
True to load models, anything else not to.
--train TRAIN True to train models, anything else not to.
--save_suffix SAVE_SUFFIX
Suffix for saving the file.
--load_suffix LOAD_SUFFIX
Suffix for loading the file.
To train a set of MADDPG agents for 1000 episodes in a formation control with collision avoidance scenario, you can run the following command
python decentralized_experiment.py
Further customization of the experiment and the agents are available through the options
usage: decentralized_experiment.py [-h] [--agent AGENT] [--scenario SCENARIO_NAME] [--num_eps NUM_EPS] [--save_images IMAGES] [--save_models SAVE_MODEL] [--load_models LOAD_MODEL]
[--train TRAIN] [--save_suffix SAVE_SUFFIX] [--load_suffix LOAD_SUFFIX]
File to run experiments for som scenario with a centralized agent.
optional arguments:
-h, --help show this help message and exit
--agent AGENT Name of the agent types: Valid values are 'decddpg' and 'maddpg'
--scenario SCENARIO_NAME
Name of the scenario we want to run: simple_formation or formation_w_goal or formation_w_coll_avoidance
--num_eps NUM_EPS Number of episodes to train for.
--save_images IMAGES True to save images and gifs, False not to.
--save_models SAVE_MODEL
True to save models, False not to.
--load_models LOAD_MODEL
True to load models, anything not to.
--train TRAIN True to train models, anything else not to.
--save_suffix SAVE_SUFFIX
Suffix for saving the file
--load_suffix LOAD_SUFFIX
Suffix for loading the file
To evaluate either agent after training, simply set the save_images option for either script to 'True'. For an example, with the decentralized agent
python decentralized_experiment.py --save_images True
Alternatively, if you wish to evaluate an agent with saved agent models, one can load the agents given the loading suffix, if one was used when saving the agents, and by setting the training option to false. For example
python decentralized_experiment.py --save_images True --load_models True --load_suffix <SUFFIX-USED-WHEN-SAVING-MODEL> --train False
Below, we include the training curves for the different agents on the formation control with collision avoidance and a goal scenario.
Find the images for the behavioural analysis of the MADDPG agents below.