Please refer to or directly use conda to install the environment using environment.yml.
conda env create -f environment.yml
To start training, simply run train.py. Inside, you will find Config parameters for both the ViT model and the LLM model, which can be fine-tuned as needed.
The main program structure and inspiration are based on minigpt-v2.